Information processing apparatus, information processing method, and recording medium

ABSTRACT

Even under a situation where an object to be presented has movement, the object can be presented as display information in a more favorable mode. An information processing apparatus includes an acquisition unit (111) that acquires information regarding movement of an object, and a control unit (111) that projects the object in a display region at a projection timing set according to a first period, and corrects display information according to a result of the projection in accordance with a plurality of display timings each set for each second period shorter than the first period, and the control unit controls correction of second display information to maintain continuity according to the movement of the object between first display information displayed according to a first projection result of the object in accordance with a first display timing, and the second display information displayed according to a second projection result of the object in accordance with a second display timing immediately after the first display timing.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a National Stage Patent Application of PCTInternational Patent Application No. PCT/JP2019/039610 (filed on Oct. 8,2019) under 35 U.S.C. § 371, which claims priority to Japanese PatentApplication No. 2018-195198 (filed on Oct. 16, 2018), which are allhereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus,an information processing method, and a recording medium.

BACKGROUND ART

In recent years, the advancement of image recognition technology hasenabled recognition of the position and orientation of a real object(that is, an object in a real space) included in an image captured by animaging device. As one of applications of such object recognition, thereis a technology called augmented reality (AR). By using the ARtechnology, virtual content (hereinafter, referred to as “virtualobject”) in various modes such as text, icons, and animations can besuperimposed on an object in the real space (hereinafter referred to as“real object”) and a superimposed image can be presented to a user. Forexample, Patent Document 1 discloses an example of a technology ofpresenting virtual content to a user using the AR technology.

An example of a method of presenting information to the user using theAR technology includes a method of using a so-called transmission-typedisplay. Specifically, the transmission-type display is supported infront of the user's eyes, and display information is presented on thedisplay, so that the user can visually recognize an image in which thedisplay information is superimposed on an optical image of the realspace.

CITATION LIST Patent Document

-   Patent Document 1: International Publication No. 2017/183346

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

By the way, in a case where an object to be presented hasthree-dimensional information, the object is presented astwo-dimensional display information as the object is projected on ascreen surface according to the position or orientation of a viewpoint,for example, and a result of the projection is drawn. However, asituation where a delay occurs in processing regarding the projectionand drawing, and the position or orientation of the viewpoint changesduring the processing is assumed. Under such a situation, there may be agap between actually presented display information and an originallyassumed presentation mode of the display information. In particular,under a situation where the object to be presented autonomously moves, auser may perceive that the display information is blurred when theobject is presented as the display information.

Therefore, the present disclosure proposes a technique of presenting anobject to be presented as display information in a favorable mode evenunder a situation where the object has movement.

Solutions to Problems

According to the present disclosure, there is provided an informationprocessing apparatus including an acquisition unit configured to acquireinformation regarding movement of an object, and a control unitconfigured to project the object in a display region at a projectiontiming set according to a first period, and correct display informationaccording to a result of the projection in accordance with a pluralityof display timings each set for each second period shorter than thefirst period, in which the control unit controls correction of seconddisplay information to maintain continuity according to the movement ofthe object between first display information displayed according to afirst projection result of the object in accordance with a first displaytiming, and the second display information displayed according to asecond projection result of the object in accordance with a seconddisplay timing immediately after the first display timing.

Furthermore, according to the present disclosure, there is provided aninformation processing method including, by a computer, acquiringinformation regarding movement of an object, and projecting the objectin a display region at a projection timing set according to a firstperiod, and correcting display information according to a result of theprojection in accordance with a plurality of display timings each setfor each second period shorter than the first period, in whichcorrection of second display information is controlled to maintaincontinuity according to the movement of the object between first displayinformation displayed according to a first projection result of theobject in accordance with a first display timing, and the second displayinformation displayed according to a second projection result of theobject in accordance with a second display timing immediately after thefirst display timing.

Furthermore, according to the present disclosure, there is provided astorage medium recording a program for causing a computer to executeacquiring information regarding movement of an object, and projectingthe object in a display region at a projection timing set according to afirst period, and correcting display information according to a resultof the projection in accordance with a plurality of display timings eachset for each second period shorter than the first period, in whichcorrection of second display information is controlled to maintaincontinuity according to the movement of the object between first displayinformation displayed according to a first projection result of theobject in accordance with a first display timing, and the second displayinformation displayed according to a second projection result of theobject in accordance with a second display timing immediately after thefirst display timing.

Effect of the Invention

As described above, according to the present disclosure, a technique ofpresenting an object to be presented as display information in afavorable mode even under a situation where the object has movement isprovided.

Note that the above-described effect is not necessarily restrictive, andany one of effects described in the present specification or any anothereffect obtainable from the present specification may be exhibited inaddition to or in place of the above-described effect.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory view for describing an example of a schematicconfiguration of an information processing system according to anembodiment of the present disclosure.

FIG. 2 is an explanatory view for describing an example of a schematicconfiguration of an input/output device according to the presentembodiment.

FIG. 3 is an explanatory diagram for describing an outline of an exampleof processing for presenting an object having three-dimensional shapeinformation as two-dimensional display information.

FIG. 4 is an explanatory diagram for describing an outline ofreprojection.

FIG. 5 is an explanatory diagram for describing an outline of amechanism in which a presented image is blurred according to a load onprocessing regarding projection and drawing of an object.

FIG. 6 is an explanatory diagram for describing an example of anoperation regarding presentation of information by an informationprocessing system according to a comparative example.

FIG. 7 is an explanatory diagram for describing a basic idea oftechnical characteristics of the information processing system accordingto the embodiment.

FIG. 8 is an explanatory diagram for describing a basic idea oftechnical characteristics of the information processing system accordingto the embodiment.

FIG. 9 is an explanatory diagram for describing an outline of technicalcharacteristics of the information processing system according to theembodiment.

FIG. 10 is a block diagram illustrating an example of a functionalconfiguration of the information processing system according to theembodiment.

FIG. 11 is a flowchart illustrating an example of a flow of a series ofprocessing of the information processing system 1 according to theembodiment.

FIG. 12 is a functional block diagram illustrating an example of ahardware configuration of an information processing apparatusconfiguring the information processing system according to theembodiment of the present disclosure.

FIG. 13 is a functional block diagram illustrating an example of ahardware configuration in a case where the information processingapparatus configuring the information processing system according to theembodiment of the present disclosure is implemented as a chip.

MODE FOR CARRYING OUT THE INVENTION

A favorable embodiment of the present disclosure will be described indetail with reference to the appended drawings. Note that, in thepresent specification and drawings, redundant description ofconfiguration elements having substantially the same functionalconfiguration is omitted by providing the same sign.

Note that the description will be given in the following order.

1. Outline

1.1. Schematic Configuration

1.2. Configuration of Input/Output Device

1.3. Principle of Self-position Estimation

1.4. Drawing Object

2. Examination of Delay Compensation

3. Technical Characteristics

3.1. Basic Idea

3.2. Functional Configuration

3.3. Processing

3.4. Modification

4. Hardware Configuration

4.1. Configuration Example as Independently Operable Device

4.2. Configuration Example When Implementing Information ProcessingApparatus as Chip

5. Conclusion

1. Outline

<1.1. Schematic Configuration>

First, an example of a schematic configuration of an informationprocessing system according to an embodiment of the present disclosurewill be described with reference to FIG. 1 . FIG. 1 is an explanatoryview for describing an example of a schematic configuration of aninformation processing system according to an embodiment of the presentdisclosure. In FIG. 1 , reference code M11 schematically represents anobject (that is, a real object) located in a real space. Furthermore,reference codes V13 and V15 schematically represent virtual content(that is, virtual objects) presented to be superimposed in the realspace. In other words, an information processing system 1 according tothe present embodiment superimposes the virtual objects on an object inthe real space such as the real object M11 on the basis of a so-calledaugmented reality (AR) technology, for example, and presents thesuperimposed objects to a user. Note that, in FIG. 1 , both the realobject and the virtual objects are presented for easy understanding ofcharacteristics of the information processing system according to thepresent embodiment.

As illustrated in FIG. 1 , an information processing system 1 accordingto the present embodiment includes an information processing apparatus10 and an input/output device 20. The information processing apparatus10 and the input/output device 20 are able to transmit and receiveinformation to and from each other via a predetermined network. Notethat the type of network connecting the information processing apparatus10 and the input/output device 20 is not particularly limited. As aspecific example, the network may be configured by a so-called wirelessnetwork such as a network based on a Wi-Fi (registered trademark)standard. Furthermore, as another example, the network may be configuredby the Internet, a dedicated line, a local area network (LAN), a widearea network (WAN), or the like. Furthermore, the network may include aplurality of networks, and some of the networks may be configured as awired network.

The input/output device 20 is a configuration for obtaining varioustypes of input information and presenting various types of outputinformation to the user who holds the input/output device 20.Furthermore, the presentation of the output information by theinput/output device 20 is controlled by the information processingapparatus 10 on the basis of the input information acquired by theinput/output device 20. For example, the input/output device 20acquires, as the input information, information for recognizing the realobject M11, and outputs the acquired information to the informationprocessing apparatus 10. The information processing apparatus 10recognizes the position of the real object M11 in the real space on thebasis of the information acquired from the input/output device 20, andcauses the input/output device 20 to present the virtual objects V13 andV15 on the basis of the recognition result. With such control, theinput/output device 20 can present, to the user, the virtual objects V13and V15 such that the virtual objects V13 and V15 are superimposed onthe real object M11 on the basis of the so-called AR technology. Notethat, in FIG. 1 , the input/output device 20 and the informationprocessing apparatus 10 are illustrated as devices different from eachother. However, the input/output device 20 and the informationprocessing apparatus 10 may be integrally configured. Furthermore,details of the configurations and processing of the input/output device20 and the information processing apparatus 10 will be separatelydescribed below.

An example of the schematic configuration of the information processingsystem according to the embodiment of the present disclosure has beendescribed with reference to FIG. 1 .

<1.2. Configuration of Input/Output Device>

Next, an example of a schematic configuration of the input/output device20 according to the present embodiment illustrated in FIG. 1 will bedescribed with reference to FIG. 2 . FIG. 2 is an explanatory view fordescribing an example of a schematic configuration of the input/outputdevice according to the present embodiment.

The input/output device 20 according to the present embodiment isconfigured as a so-called head-mounted device mounted on at least partof the head of the user and used by the user. For example, in theexample illustrated in FIG. 2 , the input/output device 20 is configuredas a so-called eyewear-type (glasses-type) device. At least one of thelens 293 a or 293 b is configured as a transmission-type display (outputunit 211). Furthermore, the input/output device 20 includes firstimaging units 201 a and 201 b, second imaging units 203 a and 203 b, anoperation unit 207, and a holding unit 291 corresponding to a frame ofthe glasses. The holding unit 291 holds the output unit 211, the firstimaging units 201 a and 201 b, the second imaging units 203 a and 203 b,and the operation unit 207 to have a predetermined positionalrelationship with respect to the head of the user when the input/outputdevice 20 is mounted on the head of the user. Furthermore, although notillustrated in FIG. 2 , the input/output device 20 may be provided witha sound collection unit for collecting a voice of the user.

Here, a more specific configuration of the input/output device 20 willbe described. For example, in the example illustrated in FIG. 2 , thelens 293 a corresponds to a lens on a right eye side, and the lens 293 bcorresponds to a lens on a left eye side. In other words, the holdingunit 291 holds the output unit 211 such that the output unit 211 (inother words, the lenses 293 a and 293 b) is located in front of the eyesof the user in the case where the input/output device 20 is worn by theuser.

The first imaging units 201 a and 201 b are configured as so-calledstereo cameras and are held by the holding unit 291 to face a directionin which the head of the user is directed (in other words, the front ofthe user) when the input/output device 20 is mounted on the head of theuser. At this time, the first imaging unit 201 a is held near the user'sright eye, and the first imaging unit 201 b is held near the user's lefteye. The first imaging units 201 a and 201 b capture a subject (in otherwords, the real object located in the real space) located in front ofthe input/output device 20 from different positions from each other onthe basis of such a configuration. Thereby, the input/output device 20acquires images of the subject located in front of the user and cancalculate a distance to the subject from the input/output device 20 onthe basis of a parallax between the images respectively captured by thefirst imaging units 201 a and 201 b. Note that, in the presentdisclosure, description of “image” can include “still image” and “movingimage” unless otherwise specified.

Note that the configuration and method are not particularly limited aslong as the distance between the input/output device 20 and the subjectcan be measured. As a specific example, the distance between theinput/output device 20 and the subject may be measured on the basis of amethod such as multi-camera stereo, moving parallax, time of flight(TOF), or structured light. Here, the TOF is a method of obtaining animage (so-called distance image) including a distance (depth) to asubject on the basis of a measurement result by projecting light such asinfrared light on the subject and measuring a time required for theprojected light to be reflected by the subject and return, for eachpixel. Furthermore, the structured light is a method of obtaining adistance image including a distance (depth) to a subject on the basis ofa change in a pattern obtained from an imaging result by irradiating thesubject with the pattern of light such as infrared light and capturingthe pattern. Furthermore, the moving parallax is a method of measuring adistance to a subject on the basis of a parallax even in a so-calledmonocular camera. Specifically, the subject is captured from differentviewpoints from each other by moving the camera, and the distance to thesubject is measured on the basis of the parallax between the capturedimages. Note that, at this time, the distance to be subject can bemeasured with more accuracy by recognizing a moving distance and amoving direction of the camera using various sensors. Note that theconfiguration of the imaging unit (for example, the monocular camera,the stereo camera, or the like) may be changed according to the distancemeasuring method.

Furthermore, the second imaging units 203 a and 203 b are held by theholding unit 291 such that eyeballs of the user are located withinrespective imaging ranges when the input/output device 20 is mounted onthe head of the user. As a specific example, the second imaging unit 203a is held such that the user's right eye is located within the imagingrange. The direction in which the line-of-sight of the right eye isdirected can be recognized on the basis of an image of the eyeball ofthe right eye captured by the second imaging unit 203 a and a positionalrelationship between the second imaging unit 203 a and the right eye onthe basis of such a configuration. Similarly, the second imaging unit203 b is held such that the user's left eye is located within theimaging range. In other words, the direction in which the line-of-sightof the left eye is directed can be recognized on the basis of an imageof the eyeball of the left eye captured by the second imaging unit 203 band a positional relationship between the second imaging unit 203 b andthe left eye. Note that the example in FIG. 2 illustrates theconfiguration in which the input/output device 20 includes both thesecond imaging units 203 a and 203 b. However, only one of the secondimaging units 203 a and 203 b may be provided.

The operation unit 207 is a configuration for receiving an operation onthe input/output device 20 from the user. The operation unit 207 may beconfigured by, for example, an input device such as a touch panel or abutton. The operation unit 207 is held at a predetermined position ofthe input/output device 20 by the holding unit 291. For example, in theexample illustrated in FIG. 2 , the operation unit 207 is held at aposition corresponding to a temple of the glasses.

Furthermore, the input/output device 20 according to the presentembodiment may be provided with, for example, an acceleration sensor andan angular velocity sensor (gyro sensor) and may be able to detectmovement of the head of the user wearing the input/output device 20 (inother words, movement of the input/output device 20 itself). As aspecific example, the input/output device 20 may recognize a change inat least either the position or orientation of the head of the user bydetecting components in a yaw direction, a pitch direction, and a rolldirection as the movement of the head of the user.

The input/output device 20 according to the present embodiment canrecognize changes in its own position and orientation in the real spaceaccording to the movement of the head of the user on the basis of theabove configuration. Furthermore, at this time, the input/output device20 can present the virtual content (in other words, the virtual object)to the output unit 211 to superimpose the virtual content on the realobject located in the real space on the basis of the so-called ARtechnology. Note that an example of a method for the input/output device20 to estimate its own position and orientation in the real space (thatis, self-position estimation) will be described below in detail.

Note that examples of a head-mounted display (HMD) device applicable asthe input/output device 20 include a see-through HMD, a videosee-through HMD, and a retinal projection HMD.

The see-through HMD uses, for example, a half mirror or a transparentlight guide plate to hold a virtual image optical system including atransparent light guide or the like in front of the eyes of the user,and displays an image inside the virtual image optical system.Therefore, the user wearing the see-through HMD can take the externalscenery into view while viewing the image displayed inside the virtualimage optical system. With such a configuration, the see-through HMD cansuperimpose an image of the virtual object on an optical image of thereal object located in the real space according to the recognitionresult of at least one of the position or orientation of the see-throughHMD on the basis of the AR technology, for example. Note that a specificexample of the see-through HMD includes a so-called glasses-typewearable device in which a portion corresponding to a lens of glasses isconfigured as a virtual image optical system. For example, theinput/output device 20 illustrated in FIG. 2 corresponds to an exampleof the see-through HMD.

In a case where the video see-through HMD is mounted on the head or faceof the user, the video see-through HMD is mounted to cover the eyes ofthe user, and a display unit such as a display is held in front of theeyes of the user. Furthermore, the video see-through HMD includes animaging unit for capturing surrounding scenery, and causes the displayunit to display an image of the scenery in front of the user captured bythe imaging unit. With such a configuration, the user wearing the videosee-through HMD has a difficulty in directly taking the external sceneryinto view but the user can confirm the external scenery with the imagedisplayed on the display unit. Furthermore, at this time, the videosee-through HMD may superimpose the virtual object on an image of theexternal scenery according to the recognition result of at least one ofthe position or orientation of the video see-through HMD on the basis ofthe AR technology, for example.

The retinal projection HMD has a projection unit held in front of theeyes of the user, and an image is projected from the projection unittoward the eyes of the user such that the image is superimposed on theexternal scenery. More specifically, in the retinal projection HMD, animage is directly projected from the projection unit onto the retinas ofthe eyes of the user, and the image is imaged on the retinas. With sucha configuration, the user can view a clearer image even in a case wherethe user has myopia or hyperopia. Furthermore, the user wearing theretinal projection HMD can take the external scenery into view evenwhile viewing the image projected from the projection unit. With such aconfiguration, the retinal projection HMD can superimpose an image ofthe virtual object on an optical image of the real object located in thereal space according to the recognition result of at least one of theposition or orientation of the retinal projection HMD on the basis ofthe AR technology, for example.

Furthermore, an HMD called immersive HMD can also be mentioned inaddition to the above-described examples. The immersive HMD is mountedto cover the eyes of the user, and a display unit such as a display isheld in front of the eyes of the user, similarly to the videosee-through HMD. Therefore, the user wearing the immersive HMD has adifficulty in directly taking an external scenery (in other words,scenery of a real world) into view, and only an image displayed on thedisplay unit comes into view. With such a configuration, the immersiveHMD can provide an immersive feeling to the user who is viewing theimage. Therefore, the immersive HMD can be applied in a case ofpresenting information mainly based on a virtual reality (VR)technology, for example.

An example of the schematic configuration of the input/output deviceaccording to the embodiment of the present disclosure has been describedwith reference to FIG. 2 .

<1.3. Principle of Self-Position Estimation>

Next, an example of a principle of a technique for the input/outputdevice 20 to estimate its own position and orientation in the real space(that is, self-position estimation) when superimposing the virtualobject on the real object will be described.

As a specific example of the self-position estimation, the input/outputdevice 20 captures an image of a marker or the like having a known sizepresented on the real object in the real space, using an imaging unitsuch as a camera provided in the input/output device 20. Then, theinput/output device 20 estimates at least one of its own relativeposition or orientation with respect to the marker (and thus the realobject on which the marker is presented) by analyzing the capturedimage. Note that the following description will be given focusing on thecase where the input/output device 20 estimates its own position andorientation. However, the input/output device 20 may estimate only oneof its own position or orientation.

Specifically, a relative direction of the imaging unit with respect tothe marker (and thus the input/output device 20 provided with theimaging unit) can be estimated according to the direction of the marker(for example, the direction of a pattern and the like of the marker)captured in the image. Furthermore, in the case where the size of themarker is known, the distance between the marker and the imaging unit(that is, the input/output device 20 provided with the imaging unit) canbe estimated according to the size of the marker in the image. Morespecifically, when the marker is captured from a farther distance, themarker is captured smaller. Furthermore, a range in the real spacecaptured in the image at this time can be estimated on the basis of anangle of view of the imaging unit. By using the above characteristics,the distance between the marker and the imaging unit can be calculatedbackward according to the size of the marker captured in the image (inother words, a ratio occupied by the marker in the angle of view). Withthe above configuration, the input/output device 20 can estimate its ownrelative position and orientation with respect to the marker.

Furthermore, a technology so-called simultaneous localization andmapping (SLAM) may be used for the self-position estimation of theinput/output device 20. SLAM is a technology for performingself-position estimation and creation of an environmental map inparallel by using an imaging unit such as a camera, various sensors, anencoder, and the like. As a more specific example, in SLAM (inparticular, Visual SLAM), a three-dimensional shape of a captured scene(or subject) is sequentially restored on the basis of a moving imagecaptured by the imaging unit. Then, by associating a restoration resultof the captured scene with a detection result of the position andorientation of the imaging unit, creation of a map of a surroundingenvironment and estimation of the position and orientation of theimaging unit (and thus the input/output device 20) in the environmentare performed. Note that the position and orientation of the imagingunit can be estimated as information indicating relative change on thebasis of detection results of various sensors by providing the varioussensors such as an acceleration sensor and an angular velocity sensor tothe input/output device 20, for example. Of course, the estimationmethod is not necessarily limited to the method based on the detectionresults of the various sensors such as an acceleration sensor and anangular velocity sensor as long as the position and orientation of theimaging unit can be estimated.

Under the above configuration, the estimation result of the relativeposition and orientation of the input/output device 20 with respect tothe known marker, which is based on the imaging result of the marker bythe imaging unit, may be used for initialization processing or positioncorrection in SLAM described above, for example. With the configuration,the input/output device 20 can estimate its own position and orientationwith respect to the marker (and thus the real object on which the markeris presented) by the self-position estimation based on SLAM reflectingresults of the initialization and position correction executed beforeeven in a situation where the marker is not included in the angle ofview of the imaging unit.

Furthermore, the above description has been made focusing on the exampleof the case of performing the self-position estimation mainly on thebasis of the imaging result of the marker. However, a detection resultof another target other than the marker may be used for theself-position estimation as long as the detection result can be used asa reference for the self-position estimation. As a specific example, adetection result of a characteristic portion of an object (real object)in the real space, such as a shape or pattern of the object, instead ofthe marker, may be used for the initialization processing or positioncorrection in SLAM.

An example of the principle of the technique for the input/output device20 to estimate its own position and orientation in the real space (thatis, self-position estimation) when superimposing the virtual object onthe real object has been described. Note that the following descriptionwill be given on the assumption that the position and orientation of theinput/output device 20 with respect to an object (real object) in thereal space can be estimated on the basis of the above-describedprinciple, for example.

<1.4. Drawing Object>

Next, an outline of an example of processing of presenting an object(for example, a virtual object) having three-dimensional shapeinformation as two-dimensional display information will be described.

For example, FIG. 3 is an explanatory diagram for describing an outlineof an example of processing of presenting an object havingthree-dimensional shape information as two-dimensional displayinformation. Note that the processing illustrated in FIG. 3 can beimplemented by processing called “vertex shader”, for example.

As illustrated in FIG. 3 , in the case of presenting an object havingthree-dimensional shape information as two-dimensional displayinformation, a target object is projected on a screen surface definedaccording to the field of view (angle of view) from an observation point(that is, viewpoint) with respect to the observation point in athree-dimensional space. That is, the screen surface can correspond to aprojection surface. At this time, a color of the object when the objectis drawn as the two-dimensional display information may be calculatedaccording to a positional relationship between a light source defined inthe three-dimensional space and the object. The two-dimensional shape ofthe object, the color of the object, the two-dimensional position atwhich the object is presented (that is, the position on the screensurface), and the like are calculated according to the relativepositional relationship among the observation point, the object, and thescreen surface according to the position and orientation of theobservation point.

Note that the example in FIG. 3 illustrates an example of a projectionmethod in a case where a clip surface is located on a back side of thetarget object, that is, in a case where the object is projected on thescreen surface located on the back side. In contrast, as another exampleof the projection method, there is a method in which the clip surface islocated on a front side of the target object, and the object isprojected on the screen surface located on the front side. In thefollowing description, the case where the clip surface is located on theback side of the target object will be described as an example, asillustrated in FIG. 3 . However, the object projection method is notnecessarily limited, and the case where the clip surface is located onthe front side of the object similarly works unless technicaldiscrepancies occur.

The projection result of the object obtained as described above is drawnas display information for a desired drawing region, for example. Notethat the processing regarding drawing of the display information can beimplemented by, for example, processing called “pixel shader”.

That is, as a specific example, the above-described desired drawingregion is associated with at least a part of the screen surfaceillustrated in FIG. 3 , and the projection result of the object onto theregion is drawn as display information, for example. Furthermore, thedrawing region is associated with a display region of an output unitsuch as a display, and a drawing result of the display information tothe drawing region can be presented in the display region. The drawingregion may be defined as at least a partial region of a predeterminedbuffer (for example, a frame buffer) that temporarily or permanentlyholds data such as the drawing result.

Furthermore, as another example, the display region itself of the outputunit may be used as the drawing region. In this case, the displayinformation is directly drawn in the display region corresponding to thedrawing region, so that the display information is presented in thedisplay region. Furthermore, in this case, the display region itself cancorrespond to the screen surface (projection surface).

As described above, an object having three-dimensional shape informationcan be presented in the display region of a predetermined output unit asthe two-dimensional display information according to a relativerelationship of the positions or orientations between the object and anobservation point (viewpoint).

The outline of an example of processing for presenting an object (forexample, a virtual object) having three-dimensional shape information astwo-dimensional display information has been described.

2. Examination of Delay Compensation

As described above, in a case where an object having a three-dimensionalshape is presented as two-dimensional display information, the object isprojected onto the screen surface according to the position ororientation relationship between the viewpoint and the object, and aresult of the projection is drawn.

However, a load on the processing regarding the projection and drawingmay become high, and a situation where a delay occurs in the processing,and the position or orientation of the viewpoint changes aftercompletion of the processing may be assumed. In particular, under asituation where the display information is presented by beingsuperimposed on an optical image of an object (real object) in the realspace on the basis of the AR technology, there may be a gap between aposition on which the display information is originally assumed to besuperimposed and a position on which the display information is actuallysuperimposed, for example. Therefore, under such a situation, the gap iseasily perceived by the user.

(Application of Reprojection)

In such a case, for example, there may be a gap (for example, a gap inposition, orientation, size, or the like) between the displayinformation that can be visually recognized according to the position ororientation of the viewpoint at that time (that is, the displayinformation visually recognized in an ideal state) and the actuallypresented display information. In such a case, the influence of the gapin the display information, which has been caused due to the delay, maybe reduced by correcting a presentation position, orientation (shape),size, and the like of the display information according to the positionor orientation of the viewpoint at that time in presenting the displayinformation according to a drawing result, for example. Note that,hereinafter, processing regarding such correction (for example,deformation) will also be referred to as reprojection.

For example, FIG. 4 is an explanatory diagram for describing an outlineof the reprojection, illustrating an example of a case of correcting apresentation position of the display information according to aprojection result of an object. In the example illustrated in FIG. 4 ,an example of display information presentation modes in a case where theprocessing regarding the projection or the drawing is executed at a rateof 30 Hz under a situation where information is presented to the outputunit at a refresh rate of 120 Hz. Specifically, in the exampleillustrated in FIG. 4 , as examples of the display informationpresentation mode, “ideal output”, “output without reprojection”, and“output with reprojection” are chronologically presented in a comparablemanner.

Note that, in the example illustrated in FIG. 4 , the “ideal output”illustrates a video in which the display information according to theresult of the projection or drawing is presented, in a case where theprocessing regarding the projection or drawing can be executed at therefresh rate of 120 Hz. Meanwhile, the “output without reprojection”illustrates a video in which the display information according to theresult of the projection or drawing is presented, in a case where theprocessing regarding the projection or drawing is executed at the rateof 30 Hz, and the reprojection is not applied. Furthermore, the “outputwith reprojection” illustrates a video in which the display informationaccording to the result of the projection or drawing is presented, in acase where the processing regarding the projection or drawing isexecuted at the rate of 30 Hz, and the reprojection is applied.

Furthermore, in FIG. 4 , the horizontal axis represents time, andreference codes t11 to t15 represent timings chronologically arranged inthis order. More specifically, in the example illustrated in FIG. 4 ,each of the timings t11 to t15 schematically represents each timing atwhich information is presented to the output unit at the refresh rate of120 Hz.

Furthermore, reference codes V101 to V105 schematically represent imagesrespectively presented at the timings t11 to t15 in a case where theideal output is performed. That is, in the case of the ideal output, ateach of the timings t11 to t15, a projection result of an objectaccording to a relative position or orientation relationship between theviewpoint and the object at that time is presented as the displayinformation.

In contrast, reference codes V111 and V115 schematically representimages respectively presented at the timings t11 to t15 in a case wherethe output without reprojection is performed. As can be seen bycomparing the ideal output and the output without reprojection, in thecase of the output without reprojection, only the image V111 iscontinuously presented in a period of the timings t11 to t14 where theimages V101 to V104 are sequentially presented as the ideal outputs.From the fact, in the case of the output without reprojection, even ifthe relative position or orientation relationship between the viewpointand the object changes during the period from the timings t11 to t14, itis difficult to present the display information in a mode consideringthe change.

Furthermore, reference codes V121 and V125 schematically representimages respectively presented at the timings t11 to t15 in a case wherethe output with reprojection is performed. Specifically, in the exampleillustrated in FIG. 4 , at each of the timings t12 to t14, correction ofthe presentation position according to the position or orientation ofthe viewpoint detected at the timing t11 is applied to the displayinformation according to the projection result of the object at thetiming t11, as the reprojection. That is, in the example illustrated inFIG. 4 , as can be seen by comparing the images V101 to V104 and theimages V121 to V124, correction has been performed such that thepresentation position of the display information according to theprojection result of the object moves toward the case where the idealoutput is performed with the execution of the reprojection. As a result,a video expressing the movement of the object with higher resolutionthan the case of the output without reprojection (that is, a video inwhich the movement of the object is more smoothly reproduced) can bepresented to the user.

(Technical Problems that Become Apparent by Applying Reprojection)

Meanwhile, in a case of reproducing a situation where an objectautonomously moves in the real space by animation or the like, the usermay perceive that a presented video is blurred depending on a load onthe processing regarding projection and drawing of the object, forexample.

For example, FIG. 5 is an explanatory diagram for describing an outlineof a mechanism in which a presented video is blurred according to a loadon the processing regarding projection and drawing of an object. In FIG.5 , reference codes t201 to t206 represent timings on a time axis in thereal space. More specifically, each of the timings t201 to t205schematically illustrates a timing at which processing (Rendering)regarding projection and drawing of an object, or processing(Displaying) regarding presentation of the display information accordingto result of the projection and drawing is executed. In contrast,reference codes t211 to t216 represent timings on a time axis on ananimation (in other words, a time axis in a virtual space) in a case ofpresenting the display information as the animation. Note that, in theexample illustrated in FIG. 5 , the timings t211 to t216 are set inassociation with the timings t201 to t206.

Furthermore, each of reference codes F101 to F104 schematicallyrepresents a situation in which the processing (Rendering) regardingprojection and drawing of the object is sequentially executed inchronological order. Furthermore, each of reference codes F111 to F114schematically represents a situation in which the processing(Displaying) regarding presentation of the display information accordingto the result of the projection and drawing is sequentially executed inchronological order. That is, a length in a horizontal direction of eachof the processing F101 to F104 and F111 to F114 schematicallyillustrates a period from start to end of the processing. That is, inthe example illustrated in FIG. 5 , the processing F102 and theprocessing F104 have a longer time from the start to the end of theprocessing than the processing F101 and the processing F103 due to theinfluence of, for example, the processing load. That is, each of thetimings t202 to t203 is set so that the period of the timings t202 tot203 and the period of the timings t204 to t205 become longer than theperiod of the timings t201 to t202 and the period of the timings t203 tot204.

For example, presentation of the display information according to theresult of the processing regarding projection and drawing of the objectis executed after the processing is terminated under the above-describedassumption. For example, in the example illustrated in FIG. 5 , theprocessing F101 regarding projection and drawing of the object isexecuted from the timing t201 as a starting point according to theposition or orientation of the viewpoint at the timing t201. Next, theprocessing F111 regarding presentation of the display informationaccording to the result of the processing F101 is executed from thetiming t202 on or after the end of the processing F101 as the startingpoint. Furthermore, when the processing F101 ends, the processing F102regarding projection and drawing of the object is executed from thetiming t202 as the starting point according to the position ororientation of the viewpoint at the timing t202 on or after the end ofthe processing F101. As described above, the processing F101 to F104regarding projection and drawing of the object and the processing F111to F114 regarding presentation of the display information according tothe results of the processing F101 to F104 are sequentially executedalong the time axis in the real space.

Furthermore, each of reference codes F121 to F124 schematicallyrepresents a situation in which an animation that reproduces movement ofthe object is presented along the time axis on the animation accordingto the execution results of the processing F111 to F114. At this time,in each of the animations F121 to F124, the movement of the object isdetermined according to the position or orientation of the viewpoint atthe timings t201 to t204. Therefore, the starting point of the movementof the object presented as each of the animations F121 to F124 is eachof timings t211 to t214 on the time axis on the animation correspondingto the timings t201 to t204. In contrast, the period during which eachof the animations F121 to F124 is presented depends on the period of thecorresponding processing among the processing F111 to F114 regardingpresentation of the display information.

More specifically, as illustrated in FIG. 5 , the animation F121 ispresented with the execution of the processing F111. At this time, themovement of the object presented as the animation F121 is determinedaccording to the position and orientation of the viewpoint at the timingt201. In other words, the movement of the object can be determinedaccording to the position or orientation relationship between theviewpoint and the object at the timing t201. Therefore, the startingpoint of the movement of the object presented as the animation F121 isthe timing t211 on the time axis on the animation corresponding to thetiming t201. Furthermore, the period during which the animation F121 ispresented depends on the period during which the processing F111 isexecuted, that is, the length of the period between the timings t202 andt203.

Next, the animation F122 is presented with the execution of theprocessing F112. At this time, the movement of the object presented asthe animation F122 is determined according to the position andorientation of the viewpoint at timing t202. Therefore, the startingpoint of the movement of the object presented as the animation F122 isthe timing t212 on the time axis on the animation corresponding to thetiming t202. Furthermore, the period during which the animation F122 ispresented depends on the period during which the processing F112 isexecuted, that is, the length of the period between the timings t203 andt204.

Next, the animation F123 is presented with the execution of theprocessing F113. At this time, the movement of the object presented asthe animation F123 is determined according to the position andorientation of the viewpoint at timing t203. Therefore, the startingpoint of the movement of the object presented as the animation F123 isthe timing t213 on the time axis on the animation corresponding to thetiming t203. Furthermore, the period during which the animation F123 ispresented depends on the period during which the processing F113 isexecuted, that is, the length of the period between the timings t204 andt205.

Furthermore, the animation F124 is presented with the execution of theprocessing F114. At this time, the movement of the object presented asthe animation F123 is determined according to the position andorientation of the viewpoint at timing t204. Therefore, the startingpoint of the movement of the object presented as the animation F124 isthe timing t214 on the time axis on the animation corresponding to thetiming t204. Furthermore, the period during which the animation F124 ispresented depends on the length of the period during which theprocessing F113 is executed, that is, the length of the period betweenthe timings t205 and t206.

Here, first, attention is paid to the relationship between thepresentation of the animation F121 and the presentation of the animationF122. In the example illustrated in FIG. 5 , the timings as the startingpoints of the movement of the object presented by the animations F121and F122 are the timings t211 and t212 as described above. Meanwhile,the period from the start to the end of the antecedently displayedanimation F121 is longer than the period between the timings t211 andt212. Therefore, as illustrated in FIG. 5 , the timing at which theanimation F121 ends is later than the timing t212 on the time axis onthe animation. Therefore, for example, in a case where correction of thedisplay information (reprojection or reprojection processing) is appliedin consideration of the length of the period between the timings t211and t212 in which the processing F111 is executed, chronologicalconnection on the time axis on the animation may become discontinuousbetween the presentation of the animation F121 and the presentation ofthe animation F122. As a specific example, when the animations F121 andF122 are displayed in sequence, the display information is sometimesdisplayed such that the object behaves as if the time went back betweenthe animations F121 and F122. This similarly applies to the relationshipbetween the presentation of the animation F123 and the presentation ofthe animation F124.

Next, attention is paid to the relationship between the presentation ofthe animation F122 and the presentation of the animation F123. In theexample illustrated in FIG. 5 , the timings as the starting points ofmovement of the object presented by the animations F122 and F123 are thetimings t212 and t213 as described above. Meanwhile, the period from thestart to the end of the antecedently displayed animation F122 is shorterthan the period between the timings t212 and t213. Therefore, the timingat which the animation F123 is started is further later than the timingat which the animation F122 ends. Therefore, for example, in a casewhere correction of the display information (reprojection) is applied inconsideration of the length of the period between the timings t212 andt213 in which the processing F112 is executed, chronological connectionon the time axis on the animation may become discontinuous between thepresentation of the animation F122 and the presentation of the animationF123. As a specific example, when the animations F121 and F122 aredisplayed in sequence, the display information is sometimes displayedsuch that the object behaves as if the time jumped between theanimations F121 and F122.

That is, as described above, when the display information is presentedas if the time went back or jumped, there is a portion where themovement of the object presented as the display information isdiscontinuous. Under such a situation, for example, the user mayperceive that the movement of the object presented as the displayinformation is blurred.

In view of the above situation, the present disclosure proposes atechnique of presenting an object to be presented as display informationin a favorable mode even under a situation where the object hasmovement. Specifically, proposed is a technique of suppressing asituation in which the display information is presented as if a portionwhere the movement of the object is discontinuous has occurred frombecoming apparent, and presenting a video in which the movement of theobject is shown in a more natural manner, in presenting the objecthaving movement as the display information.

3. Technical Characteristics

Hereinafter, technical characteristics of the information processingsystem 1 according to the embodiment of the present disclosure will bedescribed.

<3.1. Basic Idea>

First, a basic idea of the technique in which the information processingsystem 1 according to an embodiment of the present suppresses asituation in which the display information is presented as if a portionwhere the movement of the object is discontinuous has occurred frombecoming apparent, will be described below. Note that, in the presentdescription, to make characteristics of the information processingsystem according to the present embodiment easier to understand, first,an example of a case in which a portion where movement of an object isdiscontinuous occurs will be described as a comparative example, andthen the basic idea of the technical characteristics of the informationprocessing system will be described.

(Comparative Example)

For example, FIG. 6 is an explanatory diagram for describing an exampleof an operation regarding presentation of information by an informationprocessing system according to the comparative example, illustrating anexample of a case in which a portion where movement of an object isdiscontinuous occurs. Note that, in FIG. 6 , the horizontal axisrepresents “time”.

In FIG. 6 , “ts” represents each of timings on a reference time axis ofprocessing, with an identifier as a timestamp, in executing varioustypes of processing for presenting a target object as displayinformation. More specifically, in the example illustrated in FIG. 6 ,the timestamp ts represents each timing of each predetermined period(for example, each frame) on the reference time axis, using a numericalvalue as the identifier. Furthermore, the timestamp ts represents atiming of each predetermined unit period (hereinafter also referred toas “frame”) on the reference time axis. As a more specific example, inthe example illustrated in FIG. 6 , the timestamp ts=1 represents astart timing of the first frame on the reference time axis. Note that,in the following description, simple description of the “reference timeaxis” indicates a reference time axis of the various types of processingfor presenting the target object as display information unless otherwisespecified. Furthermore, in the following description, description of the“frame” indicates the predetermined unit period on the reference timeaxis unless otherwise specified. Furthermore, the predetermined unitperiod (for example, the above frame) on the reference time axiscorresponds to an example of “first period”.

“AP” schematically represents processing that can be executed by anupper layer such as an application among the various types of processingfor presenting the target object as the display information, forexample. The application in the present disclosure includes at least anapplication using augmented reality technology (AR technology), that is,an augmented reality application (AR application). The processingcorresponds to, for example, processing regarding determination andupdate of the positional relationship and movement of the target object(so-called processing regarding determination and update of a scenegraph), for example. That is, the frame corresponds to a periodregarding determination and update of a scene graph of the ARapplication. For example, reference codes F201, F202, F203, and F205schematically represent the processing that can be executed by an upperlayer for each frame. Specifically, the processing F201 represents theprocessing that can be executed by an upper layer in the framecorresponding to the timestamp ts=1 (for example, the processingregarding determination and update of a scene graph). Similarly, each ofthe processing F202, F203, and F205 represents processing that can beperformed by an upper layer in the frames respectively corresponding tothe timestamps ts=2, ts=3, and ts=5. Note that, in the followingdescription, the processing illustrated as “AP” mainly represents theprocessing regarding determination and update of a scene graph forconvenience.

“GPU” schematically represents processing that can be executed by animage processing engine such as a GPU among the various types ofprocessing for presenting the target object as the display information,for example. The processing may correspond to, for example, processingregarding projection of the object according to a positionalrelationship between a viewpoint and the object, and processingregarding drawing of display information according to a result of theprojection. For example, each of reference codes F211, F212, F213, andF215 schematically represents the processing that can be executed by theGPU in response to an execution result of the processing by an upperlayer. Specifically, the processing F211 represents the processing thatcan be executed by the GPU in response to the execution result of theprocessing F201. Similarly, each of the processing F212, F213, and F215represents processing that can be executed by the GPU in response to theexecution result of the processing F202, F203, and F205. Note that, inthe following description, the processing illustrated as “GPU” mainlyindicates the processing regarding projection of an object according tothe result of the determination or update of a scene graph (that is, theprocessing regarding projection according to the positional relationshipbetween the viewpoint and the object), and the processing regardingdrawing of the display information according to the result of theprojection. Furthermore, a timing at which the processing regardingprojection of an object is executed, among the timings at which theprocessing illustrated as “GPU” (for example, the processing F212, F213,and F215 in FIG. 6 ) is executed, especially corresponds to an exampleof “projection timing”.

“Reprojection” schematically represents processing corresponding toreprojection, in other words, processing regarding correction of thedisplay information according to a drawing result. For example, each ofreference codes F2211 and F2212 represents processing corresponding toreprojection executed according to the position or orientation of theviewpoint in accordance with a presentation timing of the displayinformation in response to the execution result of the processing F211.Similarly, each of reference codes F2221 to F2224 represents processingcorresponding to reprojection executed in response to the executionresult of the processing F212. Furthermore, each of reference codesF2231 to F2232 represents processing corresponding to reprojectionexecuted in response to the execution result of the processing F213.Furthermore, reference code F2251 represents processing corresponding toreprojection executed in response to the execution result of theprocessing F215.

Note that, in the example illustrated in FIG. 6 , the processingillustrated AP and GPU is assumed to be executed at a rate of 60 Hz.Furthermore, as for the processing illustrated as Reprojection is setaccording to, for example, a refresh rate of an output unit. In theexample illustrated in FIG. 6 , the processing is assumed to be executedat the rate of 120 Hz. In other words, the processing corresponding tothe reprojection can be executed at the timing (that is, display timing)when the display information is presented via the output unit. Note thata period during which the display information is continuously presented(that is, a period between the display timings adjacent to each other onthe time axis) corresponds to an example of “second period”. In otherwords, the processing corresponding to the reprojection can be executedevery second period. Furthermore, the second period is set to be shorterthan the above-mentioned first period (that is, the above-describedframe). Furthermore, in the example illustrated in FIG. 6 , a timing atwhich each processing is started is set at the rate of 120 Hz (that is,the timing is set as a timing of every 0.5 frames corresponding to thesecond period). In other words, the timing at which each processing isstarted between the timestamp ts=n and the timestamp ts=n+1 is a timingof one of the timestamps ts=n, ts=n+0.5, and ts=n+1.

Here, an outline of the series of processing illustrated as AP, GPU, andReprojection will be described by taking processing executed from thetimestamp ts=1 as a starting point, as an example. Specifically, theprocessing F201 is executed from the timing corresponding to thetimestamp ts=1 as a starting point, and a scene graph corresponding tothe timestamp ts=1 is determined or updated.

Note that, in the following description, a timing corresponding to thetimestamp ts=n among timings at which a target scene is determined orupdate is illustrated as “Scene:n”. That is, in the case where a sceneis determined or updated at the timing corresponding to the timestampts=n, the scene graph corresponds to a scene graph corresponding to thetiming “Scene:n”. That is, the scene graph corresponding to the timing“Scene:1” can be obtained with execution of the processing F201.

Next, the processing F211 is executed in response to the executionresult of the processing F201. That is, projection of the objectaccording to the scene graph corresponding to the timing “Scene: 1” anddrawing of the display information according to the result of theprojection are executed. Note that the execution timing of theprocessing F211 is a timing corresponding to the timestamp ts=1.5immediately after being delayed by at least a period associated with theexecution of the processing F201 from the timing corresponding to thetimestamp ts=1 among the timings of every 0.5 frames. Therefore, at thetime of executing the processing F211, the object is projected accordingto the position or orientation of the viewpoint acquired at the timingcorresponding to the timestamp ts=1.5 and the display informationaccording to the result of the projection is drawn.

Note that, in the following description, a timing corresponding to thetimestamp ts=n among timings at which the position or orientation of theviewpoint is determined with the execution of reprojection isillustrated as “View:n”. That is, in a case where display informationcan be obtained as the results of the projection and the drawingaccording to the position or orientation of the viewpoint at the timingcorresponding to the timestamp ts=n, the display information correspondsto display information corresponding to the timing “View:n”. That is, inthe processing F211, the display information corresponding to the timing“View:1.5” is obtained as a result of the projection or drawingaccording to the position or orientation of the viewpoint correspondingto the timestamp ts=1.5 on the basis of the scene graph corresponding tothe timing “Scene:1”.

Next, the processing F2211 corresponding to the reprojection is executedin response to the execution result of the processing F211. At thistime, the execution timing of the processing F2211 is a timingcorresponding to the timestamp ts=2 immediately after being delayed byat least a period associated with the execution of the processing F211from the timing corresponding to the timestamp ts=1.5 at which executionof the processing F211 is started among the timings of every 0.5 frames.Therefore, in the processing F2211, the display information is correctedaccording to the execution result of the processing F211 inconsideration of the delay of 0.5 frames due to the execution of theprocessing F211. That is, in the processing F2211, a scene graphcorresponding to the timing “Scene:1.5” is assumed, and the displayinformation is corrected according to the position or orientation of theviewpoint corresponding to the timestamp ts=2, whereby the displayinformation corresponding to the timing “View:2” is obtained. That is,in this case, the display information corresponding to the timing“View:2” is presented to the user via the output unit.

Furthermore, at the timing 0.5 frames after the execution of theprocessing F2211, a new drawing result (that is, an execution result ofthe processing F212 executed after the processing F211) has not beenobtained. Therefore, when the processing F2212 is executed, informationbetween the display information according to the execution result by theprocessing F211 and the display information according to the executionresult by the processing F212 is complemented. At this time, theexecution timing of the processing F2212 is a timing corresponding tothe timestamp ts=2.5 delayed by 0.5 frames from the timing correspondingto the timestamp ts=2 at which execution of the processing F2211 isstarted. Therefore, in the processing F2212, the scene graphcorresponding to the timing “Scene:2” is assumed, and the displayinformation is corrected according to the position or orientation of theviewpoint corresponding to the timestamp ts=2.5, whereby the displayinformation corresponding to the timing “View:2.5” is obtained. That is,in this case, the display information corresponding to the timing“View:2.5” is presented to the user via the output unit.

Next, attention is paid to the processing F203 and F213 that areexecuted from the timestamp ts=3 as a starting point. The processingF203 has a high load on the processing regarding determination of thescene graph, such as a case where movement of the object is complicated,and the processing delay becomes larger than that of the otherprocessing F201 and F202, for example. As a result, it becomes difficultto execute the processing F213 at the timing corresponding to thetimestamp ts=3.5, and the timing at which the processing F213 isexecuted is delayed to the timing corresponding to the timestamp ts=4.That is, in the processing F213, the display information correspondingto the timing “View:4” is obtained as a result of the projection and thedrawing according to the position or orientation of the viewpointcorresponding to the timestamp ts=4 on the basis of the scene graphcorresponding to the timing “Scene:3”.

Furthermore, the start timing of the processing F2231 corresponding tothe reprojection in response to the execution result of the processingF213 is also delayed to the timing corresponding to the timestamp ts=5.Therefore, in the processing F2231, the display information is correctedaccording to the execution result of the processing F213 inconsideration of the delay of 1 frame due to the execution of theprocessing F213. That is, in the processing F2231, a scene graphcorresponding to the timing “Scene:4” is assumed, and the displayinformation is corrected according to the position or orientation of theviewpoint corresponding to the timestamp ts=5, whereby the displayinformation corresponding to the timing “View:5” is obtained. That is,in this case, the display information corresponding to the timing“View:5” is presented to the user via the output unit.

Meanwhile, at the timing corresponding to the timestamp ts=4.5, theexecution of the processing F213 has not been completed, the processingF2224 is executed according to the execution result of the processingF212, and the display information according to the execution result ofthe processing F2224 is presented. Specifically, the timing at which theprocessing F2224 is executed is delayed by 2 frames from the starttiming of the processing F212. Therefore, in the processing F2224, ascene graph corresponding to the timing “Scene:4” is assumed, and thedisplay information is corrected according to the position ororientation of the viewpoint corresponding to the timestamp ts=4.5,whereby the display information corresponding to the timing “View:4.5”is obtained. That is, in this case, the display informationcorresponding to the timing “View:4.5” is presented to the user via theoutput unit.

Here, as can be seen by comparing the execution result of the processingF2224 and the execution result of the processing F2231, the displayinformation assuming the scene graph corresponding to the timing “Scene:4” is presented in both cases. That is, although continuity regardingthe presentation of the display information is guaranteed between thetiming “View:4.5” and the timing “View:5”, some scenes are not updated.Therefore, for example, the user may perceive that a portion where themovement of the object is discontinuous has occurred.

Next, attention is paid to the processing F205 and F215 that areexecuted from the timestamp ts=5 as a starting point. The execution ofthe processing F205 and the processing F215 is completed within oneframe as in the case of the processing F201 and the processing F211, andthe processing F202 and the processing F212. That is, in the processingF215, the display information corresponding to the timing “View:5.5” isobtained as a result of the projection or drawing according to theposition or orientation of the viewpoint corresponding to the timestampts=5.5 on the basis of the scene graph corresponding to the timing“Scene:5”. Furthermore, the start timing of the processing F2251corresponding to the reprojection in response to the execution result ofthe processing F215 is the timing corresponding to the timestamp ts=6.That is, in the processing F2251, a scene graph corresponding to thetiming “Scene:5.5” is assumed, and the display information is correctedaccording to the position or orientation of the viewpoint correspondingto the timestamp ts=6, whereby the display information corresponding tothe timing “View:6” is obtained. That is, in this case, the displayinformation corresponding to the timing “View:6” is presented to theuser via the output unit.

Meanwhile, at the timing corresponding to the timestamp ts=5.5, theexecution of the processing F215 has not been completed, the processingF2232 is executed according to the execution result of the processingF213, and the display information according to the execution result ofthe processing F2232 is presented. Specifically, the timing at which theprocessing F2232 is executed is delayed by 1.5 frames from the starttiming of the processing F213. Therefore, in the processing F2232, ascene graph corresponding to the timing “Scene:4.5” is assumed, and thedisplay information is corrected according to the position ororientation of the viewpoint corresponding to the timestamp ts=5.5,whereby the display information corresponding to the timing “View:5.5”is obtained. That is, in this case, the display informationcorresponding to the timing “View:5.5” is presented to the user via theoutput unit.

Here, as can be seen by comparing the execution result of the processingF2231 and the execution result of the processing F2251, the displayinformation is presented in such a manner that the scene graphcorresponding to “Scene:4.5” jumps to the scene graph corresponding to“Scene:5.5”. That is, although continuity regarding the presentation ofthe display information is guaranteed between the timing “View:5.5” andthe timing “View:6”, some scenes lack. Therefore, for example, the usermay perceive that a portion where the movement of the object isdiscontinuous has occurred.

As described above, in the information processing system according tothe comparative example, a portion where the movement of the object isdiscontinuous may occur at the timing at which pieces of the displayinformation are displayed in succession, due to the influence of anincrease in the processing load, for example.

(Example)

Next, the basic idea of the technical characteristics of the informationprocessing system according to the present embodiment will be describedusing specific examples. For example, FIG. 7 is explanatory diagrams fordescribing the basic idea of the technical characteristics of theinformation processing system according to the present embodiment. InFIG. 7 , the horizontal axis represents “time”. Furthermore, in FIG. 7 ,processing represented as “ts”, “GPU”, and “Reprojection” is similar tothe processing represented “ts”, “GPU”, and “Reprojection” illustratedin FIG. 6 . Furthermore, in FIG. 7 , processing F301, F302, F303, andF305 is similar to the processing F201, F202, F203, and F205 illustratedin FIG. 6 . Furthermore, in FIG. 7 , processing F311, F312, F313, andF315 is similar to the processing F211, F212, F213, and F215 illustratedin FIG. 6 . Therefore, detailed description of the above processingsimilar to the example illustrated in FIG. 6 will be omitted.

Furthermore, in FIG. 7 , processing F3211, F3212, F3221, F3222, F3223,F3224, F3231, F3232, and F3251 corresponds to the processing F2211,F2212, F2221, F2222, F2223, F2224, F2231, F2232, and F2251 illustratedin FIG. 6 , respectively. Note that, hereinafter, the characteristics ofthe information processing system according to the present embodimentwill be described mainly focusing on the relationship between theprocessing F3224 and the processing F3231, which corresponds to theportion where the movement of the object is discontinuous in the examplein FIG. 6 , and the relationship between the processing F3232 and theprocessing F3251. Specifically, hereinafter, the basic idea of thetechnical characteristics of the information processing system accordingto the present embodiment will be described mainly focusing on theprocessing F303, F313, F3231, and F3232, which is executed from thetimestamp ts=3 as a starting point.

In the example illustrated in FIG. 6 , in the processing F2231, only thedelay of the processing F213 regarding the projection of the object andthe drawing according to the projection result has been considered inestimating the scene graph at the time of correction (reprojection) ofthe display information. Therefore, for example, in a case where a delayoccurs in the processing F203 regarding the determination or update ofthe scene graph, there may be case where the delay is not taken intoconsideration, and as a result, the display information is presented asif a portion where the movement of the object is discontinuous hasoccurred. In contrast, in the information processing system according tothe present embodiment, the delay of the processing F303 regardingdetermination or update of a scene graph, the delay of the processingF313 regarding projection of an object according to a result of theprocessing F303 and drawing of the display information according to aresult of the projection, and the like are taken into consideration inestimating (determining) the scene graph at the time of correcting(reprojecting) the display information.

For example, FIG. 8 is an explanatory diagram for describing the basicidea of the technical characteristics of the information processingsystem according to the present embodiment, illustrating an outline of amethod of calculating the delays taken into consideration at the time ofcorrecting (reprojecting) the display information. In FIG. 8 , thehorizontal axis represents “time”. That is, FIG. 8 schematicallyillustrates a situation in which a series of processing for presentingthe target object as the display information is sequentially executedalong the time axis. Specifically, the processing illustrated as “Sceneupdate” schematically represents the processing regarding determinationand update of a scene graph, and corresponds to, for example, theprocessing illustrated as “AP” in FIG. 7 . The processing illustrated as“drawing” schematically illustrates the processing regarding projectionof the object and drawing of the display information according to aresult of the projection, and corresponds to, for example, theprocessing illustrated as “GPU” in FIG. 7 . The processing illustratedas “reprojection” schematically illustrates the processing regardingcorrection of the display information according to a result of thedrawing (that is, reprojection), and corresponds to, for example, theprocessing illustrated as “Reprojection” in FIG. 7 . Furthermore, theprocessing illustrated as “light emission” schematically illustrates theprocessing regarding presentation of the display information via theoutput unit, such as processing regarding light emission of a lightemitting body such as a backlight of a display.

Furthermore, in FIG. 8 , the period illustrated as dt schematicallyillustrates the delay taken into consideration in estimating the scenegraph at the time of correcting (reprojecting) the display information.Specifically, the timing t_scene represents the start timing of theprocessing illustrated as “Scene update”. Furthermore, the timingt_repro represents the start timing of the processing illustrated as“reprojection”. That is, in the information processing system accordingto the present embodiment, the scene graph at the time of correction isestimated in consideration of the delay of the processing illustrated as“Scene update” and “drawing” in executing the processing regardingcorrection of the display information illustrated as “reprojection”.

That is, the delay dt can be expressed by the calculation formula givenas the expression (1) below. In the following expression (1), t_sceneand t_repro are as described with reference to FIG. 8 . Furthermore,offset represents an offset that can be applied in calculating the delaydt, and can be applied in consideration of the load on the system, forexample. Note that in a case where the offset is not taken intoconsideration, the offset=0 is simply set.[Math. 1]dt=t_repro−t_scene−offset  (1)

As a specific example, in the example illustrated in FIG. 7 , in theprocessing F3231, in correcting the display information according to theexecution result of the processing F313, a scene graph at the time ofcorrection is estimated in consideration of the delay in the processingF303 and the processing F313. More specifically, there is a delay of 2frames between the timing at which execution of the processing F303 isstarted (that is, the timing corresponding to the timestamp ts=3) andthe timing at which execution of the processing F3231 is started (thetiming corresponding to the timestamp ts=5). Furthermore, in the exampleillustrated in FIG. 7 , 0.5 frames are taken into consideration as anoffset in calculating the delay dt. That is, the delay dt considered forthe estimation of the scene graph is 1.5 frames in executing theprocessing F3231. Thereby, in the processing F3231, a scene graphcorresponding to the timing “Scene:4.5” is assumed, and the displayinformation is corrected according to the position or orientation of theviewpoint corresponding to the timestamp ts=5, whereby the displayinformation corresponding to the timing “View:5” is obtained.

Furthermore, in the processing F3224, in correcting the displayinformation according to the execution result of the processing F312, ascene graph at the time of correction is estimated in consideration ofthe delay in the processing F302 and the processing F312. Morespecifically, there is a delay of 2.5 frames between the timing at whichexecution of the processing F302 is started (that is, the timingcorresponding to the timestamp ts=2) and the timing at which executionof the processing F3224 is started (the timing corresponding to thetimestamp ts=4.5). Furthermore, as described above, 0.5 frames are takeninto consideration as an offset in calculating the delay dt. That is,the delay dt considered for the estimation of the scene graph is 2frames in executing the processing F3224. Thereby, in the processingF3224, a scene graph corresponding to the timing “Scene:4” is assumed,and the display information is corrected according to the position ororientation of the viewpoint corresponding to the timestamp ts=4.5,whereby the display information corresponding to the timing “View:4.5”is obtained.

Here, as can be seen from a comparison between the execution result ofthe processing F3224 and the execution result of the processing F3231,the difference between the scene graphs is 0.5 frames, which matches 0.5frames as a difference between timings when the position or orientationof the viewpoint has been determined. From the above fact, it can beseen that the correction (reprojection) of the display information iscontrolled to maintain the continuity of the scenes between theexecution result of the processing F3224 and the execution result of theprocessing F3231 (the continuity of the movement of the object, forexample).

Furthermore, as another example, in the processing F3232, in correctingthe display information according to the execution result of theprocessing F313, a scene graph at the time of correction is estimated inconsideration of the delay in the processing F303 and the processingF313. More specifically, there is a delay of 2.5 frames between thetiming at which execution of the processing F303 is started (that is,the timing corresponding to the timestamp ts=3) and the timing at whichexecution of the processing F3232 is started (the timing correspondingto the timestamp ts=5.5). Furthermore, as described above, 0.5 framesare taken into consideration as an offset in calculating the delay dt.That is, the delay dt considered for the estimation of the scene graphis 2 frames in executing the processing F3232. Thereby, in theprocessing F3232, a scene graph corresponding to the timing “Scene:5” isassumed, and the display information is corrected according to theposition or orientation of the viewpoint corresponding to the timestampts=5.5, whereby the display information corresponding to the timing“View:5.5” is obtained.

Furthermore, in the processing F3251, in correcting the displayinformation according to the execution result of the processing F315, ascene graph at the time of correction is estimated in consideration ofthe delay in the processing F305 and the processing F315. Morespecifically, there is a delay of 1 frame between the timing at whichexecution of the processing F305 is started (that is, the timingcorresponding to the timestamp ts=5) and the timing at which executionof the processing F3251 is started (the timing corresponding to thetimestamp ts=6). Furthermore, as described above, 0.5 frames are takeninto consideration as an offset in calculating the delay dt. That is,the delay dt considered for the estimation of the scene graph is 0.5frames in executing the processing F3251. Thereby, in the processingF3251, a scene graph corresponding to the timing “Scene:5.5” is assumed,and the display information is corrected according to the position ororientation of the viewpoint corresponding to the timestamp ts=6,whereby the display information corresponding to the timing “View:6” isobtained.

Here, as can be seen from a comparison between the execution result ofthe processing F3232 and the execution result of the processing F3251,the difference between the scene graphs is 0.5 frames, which matches 0.5frames as a difference between timings when the position or orientationof the viewpoint has been determined. From the above fact, it can beseen that the correction (reprojection) of the display information iscontrolled to maintain the continuity of the scenes between theexecution result of the processing F3232 and the execution result of theprocessing F3251 (the continuity of the movement of the object, forexample).

As described above, the information processing system according to thepresent embodiment controls the correction of the display information tomaintain the continuity of the scenes among a plurality of pieces ofdisplay information according to the projection results different fromone another (for example, the continuity according to the movement ofthe object) in the case of continuously displaying the plurality ofpieces of display information.

Note that, in the case of focusing on the relationship between theprocessing F3224 and the processing F3231, the display informationdisplayed according to the antecedently executed processing F3224corresponds to an example of “first display information”, and the timingat which the first display information is displayed corresponds to anexample of “first display timing”. Furthermore, the projection result ofthe object corresponding to the processing F312, which is the correctiontarget of the processing F3224, corresponds to an example of “firstprojection result”. That is, among the execution timings of theprocessing F312, the timing at which the processing regarding theprojection of the object is executed corresponds to an example of “firstprojection timing”. Furthermore, the display information displayedaccording to the processing F3231 executed next corresponds to anexample of “second display information”, and the timing at which thefirst display information is displayed corresponds to an example of“second display timing”. In this case, the projection result of theobject corresponding to the processing F313, which is the correctiontarget of the processing F3231, corresponds to an example of “secondprojection result”. That is, among the execution timings of theprocessing F313, the timing at which the processing regarding theprojection of the object is executed corresponds to an example of the“second projection timing”. As described above, between the processingF312 and the processing F313 regarding the projection and drawing of anobject executed in sequence, the result of the projection according tothe processing F312, which is antecedently executed, corresponds to anexample of a “first projection result”, and the result of the projectionaccording to the processing F313, which is executed next, corresponds toan example of a “second projection result”. Furthermore, the “firstdisplay timing” and the “second display timing” are display timingsadjacent to each other on the time axis, and correspond to the timingsat which the “first display information” and the “second displayinformation” according to the projection results different from eachother are displayed. That is, the information processing systemaccording to the present embodiment controls the correction of thesecond display information to maintain the continuity (scene continuity)according to the movement of the object between the first displayinformation and the second display information. This similarly appliesto other examples in which a plurality of pieces of display informationaccording to the projection results different from each other iscontinuously displayed (for example, the relationship between theprocessing F3212 and F3221 and the relationship between the processingF3232 and F3251) in the example illustrated in FIG. 7 .

(Application of Correction)

Next, application of the correction of the display information by theinformation processing system according to the present embodiment willbe described using specific examples. Note that, in the presentdescription, to make the technical characteristics of the informationprocessing system according to the present embodiment easier tounderstand, a case of correcting the presentation position of thedisplay information according to the projection result of the targetobject, assuming that the object is performing constant velocity linearmotion, will be described.

For example, FIG. 9 is an explanatory diagram for describing the outlineof the technical characteristics of the information processing systemaccording to the present embodiment, illustrating an example of apresentation mode for the display information according to anapplication result of the correction of the display information.Specifically, in the example illustrated in FIG. 9 , as examples of thedisplay information presentation mode, “ideal output”, “output with agap”, and “output in which the gap has been eliminated” arechronologically presented in a comparable manner.

Note that, in the example illustrated in FIG. 9 , “ideal output” issimilar to the “ideal output” in the example described with reference toFIG. 4 . That is, in a case where the processing regarding theprojection of the object and the drawing according to a result of theprojection can be executed at the refresh rate of 120 Hz, a video inwhich the display information according to the result of the projectionand the drawing is presented is illustrated. Furthermore, the “outputwith a gap” indicates an example of a video of the case in which theuser perceives that a portion where the movement of the object isdiscontinuous has occurred, as in the example described with referenceto FIG. 6 . In contrast, the “output in which the gap has beeneliminated” indicates an example of a video in which the displayinformation is presented according to a control result of the correctionof the display information by the information processing systemaccording to the present embodiment, as in the example described withreference to FIG. 7 .

Furthermore, in FIG. 9 , the horizontal axis represents time, andreference codes t31 to t35 represent timings chronologically arranged inthis order. More specifically, in the example illustrated in FIG. 9 ,each of the timings t31 to t35 schematically represents each timing atwhich information is presented to the output unit at the refresh rate of120 Hz.

In FIG. 9 , reference codes V201 to V205 schematically represent imagesrespectively presented at the timings t31 to t35 in the case where theideal output is performed. That is, in the case of the ideal output, ateach of the timings t31 to t35, the projection result of the objectaccording to the relative position or orientation relationship betweenthe viewpoint and the object at that time is presented as the displayinformation.

Furthermore, reference codes V211 to V215 schematically represent imagesrespectively presented at the timings t31 to t35 in the case where theoutput with a gap is performed. As can be seen by comparing the outputwith a gap with the ideal output, gaps have occurred in the presentationposition of the display information at the timings t33 and t34 in thecase of the output with a gap, as compared with the case of the idealoutput. Such gaps may become apparent, for example, due to theprocessing described with reference to FIG. 6 .

In contrast, reference codes V221 to V225 schematically represent imagesrespectively presented at the timings t31 to t35 in the case where theoutput in which the gap has been eliminated is performed. As can be seenby comparing the output in which the gap has been eliminated with theideal output, the correction is controlled such that the presentationposition of the object substantially coincides with the ideal output atany of the timings t31 to t35 in the case of the output in which the gaphas been eliminated.

Note that, in the example illustrated in FIG. 9 , among the changesaccording to the movement of the object (for example, the changes in thepresentation position, orientation, size, and the like of the object),only the presentation position of the display information is correctedaccording to the position and orientation of the viewpoint of the time.Therefore, at the timings t32 to t34, for example, the orientation ofthe object does not change from the state presented as the image V221 atthe timing t31. That is, the information processing system according tothe present embodiment controls the correction of the displayinformation between the image V223 and the image V224 so as tosubstantially coincide with the change in a component related to thecorrection (that is, the change in the presentation position of thedisplay information) among the changes according to the movement of theobject between the timings t33 and t34 (in other words, the differencebetween the image V203 and the image V204), for example. Note that, inthis case, the display information presented to the image V223corresponds to an example of the “first display information”, and thedisplay information presented to the image V224 corresponds to anexample of the “second display information”. That is, in the exampleillustrated in FIG. 9 , the presentation position of the second displayinformation is determined on the basis of the first display informationfor which the reprojection processing has been performed in accordancewith the display timing (first display timing) immediately before thedisplay timing (second display timing) of the second displayinformation. In other words, the presentation position of the seconddisplay information is determined on the basis of the timestamp of thefirst display information (that is, the timing indicated by View:N) atwhich the reprojection processing has been performed.

Furthermore, in the example illustrated in FIG. 9 , an example of thecase where the presentation position of the display informationaccording to the projection result of the object is corrected accordingto the movement of the object has been described. However, the componentto be corrected is not necessarily limited. As a specific example,components such as the orientation of the display information accordingto the projection result of the object, the size of the displayinformation, the color of the display information, and the brightness ofthe display information may be component to be corrected (reprojection).As a more specific example, the orientation of the display informationmay be corrected by rotating the display information according to theprojection result of the object according to the movement of the object.Furthermore, as another example, the size of the display information maybe corrected by enlarging or reducing the display information accordingto the projection result of the object according to the movement of theobject. Furthermore, as another example, the color and brightness of thedisplay information according to the projection result of the object maybe corrected according to the movement of the object.

By the above control, the situation in which the display information ispresented as if a portion where the movement of the object isdiscontinuous has occurred as described with reference to FIGS. 5 and 6can be suppressed from becoming apparent in presenting the object withmovement as the display information. That is, the information processingsystem according to an embodiment of the present disclosure can presenta video in which the movement of an object to be presented is shown in amore natural manner, even under the situation where the object hasmovement.

As described above, the basic idea of the technique in which theinformation processing system 1 according to an embodiment of thepresent suppresses a situation in which the display information ispresented as if a portion where the movement of the object isdiscontinuous has occurred from becoming apparent, has been describedwith reference to FIGS. 6 to 9 .

<3.2. Functional Configuration>

Next, an example of a functional configuration of the informationprocessing system according to the embodiment of the present disclosurewill be described with reference to FIG. 10 . FIG. 10 is a block diagramillustrating an example of a functional configuration of the informationprocessing system according to the present embodiment. Specifically,FIG. 10 illustrates an example of a functional configuration of theinformation processing system 1 illustrated in FIG. 1 , in particular,focusing on the configuration in which the information processingapparatus 10 presents information to the user via the output unit 211 ofthe input/output device 20 according to the change in the position ororientation of the input/output device 20.

As illustrated in FIG. 10 , the information processing system 1according to the present embodiment includes an imaging unit 201, anoutput unit 211, a detection unit 251, and the information processingapparatus 10. Note that the output unit 211 corresponds to, for example,the output unit 211 described with reference to FIG. 2 .

The imaging unit 201 corresponds to the first imaging units 201 a and201 b configured as the stereo camera in FIG. 2 . The imaging unit 201captures an image of an object (subject) in the real space and outputsthe captured image to the information processing apparatus 10.

The detection unit 251 schematically illustrates a portion regardingacquisition of information for detecting a change in the position ororientation of the input/output device 20 (and thus the movement of thehead of the user wearing the input/output device 20). In other words,the detection unit 251 acquires information for detecting the change inthe position or orientation of the viewpoint (in other words, the changein the position or orientation of the input/output device 20). As aspecific example, the detection unit 251 may include various sensorsrelated to detection of movement of an object, such as an accelerationsensor and an angular velocity sensor. The detection unit 251 outputsthe acquired information to the information processing apparatus 10.Thereby, the information processing apparatus 10 can recognize thechange in the position or orientation of the input/output device 20.

Next, the configuration of the information processing apparatus 10 willbe described. As illustrated in FIG. 10 , the information processingapparatus 10 includes a recognition processing unit 101, a calculationunit 103, a drawing processing unit 105, a correction processing unit107, and an output control unit 109.

The recognition processing unit 101 acquires the image captured by theimaging unit 201, and applies analysis processing to the acquired image,thereby recognizing the object (subject) in the real space captured inthe image. As a specific example, the recognition processing unit 101acquires images captured from a plurality of different viewpoints(hereinafter also referred to as “stereo images”) from the imaging unit201 configured as a stereo camera, and measures the distance to theobject captured in the image for each pixel of the image on the basis ofthe parallax between the acquired images. Thereby, the recognitionprocessing unit 101 can estimate or recognize the relative positionalrelationship in the real space (in particular, the positionalrelationship in the depth direction) between the imaging unit 201 (andthus the input/output device 20) and each object captured in the imageat the timing at which the image is captured. The above is merely anexample, and the method and the configuration therefor are notparticularly limited as long as an object in the real space can berecognized. That is, the configuration of the imaging unit 201 and thelike may be changed as appropriate according to the method ofrecognizing an object in the real space.

Furthermore, the recognition processing unit 101 may recognize theposition or orientation of the viewpoint on the basis of the technologyof self-position estimation, for example. As a specific example, therecognition processing unit 101 may perform self-position estimation andenvironment map creation on the basis of SLAM, thereby recognizing thepositional relationship between the input/output device 20 (in otherwords, the viewpoint) and the object captured in the image in the realspace. In this case, the recognition processing unit 101 may acquireinformation regarding the detection result of the change in the positionand orientation of the input/output device 20 from the detection unit251, and use the acquired information for the self-position estimationbased on SLAM, for example. Note that the above is merely an example,and the method and the configuration therefor are not particularlylimited as long as the position or orientation of the viewpoint can berecognized. That is, the configuration of the imaging unit 201, thedetection unit 251, or the like may be changed as appropriate accordingto the method of recognizing the position or orientation of theviewpoint.

Then, the recognition processing unit 101 outputs information regardingthe result of the self-position estimation of the input/output device 20(that is, the recognition result of the position or orientation of theviewpoint) to the calculation unit 103, the drawing processing unit 105,and the correction processing unit 107 to be described below. Note that,since the information regarding the result of self-position estimationis sequentially acquired, the change in the position or orientation ofthe input/output device 20 (that is, the change in the position ororientation of the viewpoint) can be recognized, for example.

Furthermore, the recognition processing unit 101 may recognize theposition in the real space of each object (that is, the real object)captured in the image, and output information regarding the recognitionresult to the calculation unit 103, the drawing processing unit 105, andthe correction processing unit 107. As a specific example, therecognition processing unit 101 may output information (that is, depthmap) indicating the depth (the distance to the object) measured for eachpixel in the image to the calculation unit 103, the drawing processingunit 105, and the correction processing unit 107. Thereby, thecalculation unit 103, the drawing processing unit 105, and thecorrection processing unit 107 can recognize the object in the realspace on the basis of the information.

Note that the method of measuring the distance to the subject is notlimited to the above-described measuring method based on a stereo image.Therefore, the configuration corresponding to the imaging unit 201 maybe appropriately changed according to the distance measuring method. Asa specific example, in the case of measuring the distance to the subjectbased on TOF, a light source for projecting an infrared light and alight-receiving element for detecting the infrared light projected fromthe light source and reflected at the subject may be provided instead ofthe imaging unit 201. Furthermore, when measuring the distance to theobject, a plurality of measuring methods may be used. In this case, aconfiguration for acquiring information to be used for the measurementmay be provided in the input/output device 20 or the informationprocessing apparatus 10 according to the measuring method to be used. Ofcourse, it goes without saying that the content of the information (forexample, the depth map) indicating the recognition result of theposition in the real space of each object captured in the image may beappropriately changed according to the applied measuring method.

As described above, the recognition processing unit 101 sequentiallyrecognizes the position or orientation of the viewpoint, and outputs theinformation regarding the recognition result of the position ororientation of the viewpoint (in other words, information regarding themovement of the viewpoint) to the calculation unit 103, the drawingprocessing unit 105, and the correction processing unit 107 to bedescribed below.

The calculation unit 103 determines or updates the position,orientation, and movement (for example, a moving direction and speed) ofan object to be presented as the display information (for example, avirtual object) with execution of a predetermined function such as anapplication. In other words, the calculation unit 103 determines orupdates the scene graph. At this time, the calculation unit 103 mayacquire the information regarding the recognition result of the positionor orientation of the viewpoint from the recognition processing unit101, and determine or update the position, orientation, movement, andthe like of the target object on the basis of the information. As a morespecific example, the calculation unit 103 may determine or update theposition, orientation, movement, and the like of the target objectaccording to the relative position or orientation relationship betweenthe viewpoint and the object on the basis of the information acquiredfrom the recognition processing unit 101.

Furthermore, the calculation unit 103 may acquire information regardinga recognition result of an object in the real space from the recognitionprocessing unit 101, and determine or update the position, orientation,movement, and the like of the target object on the basis of theinformation. As a more specific example, the calculation unit 103 maydetermine or update the position, orientation, movement, and the like ofthe target object according to the relative position or orientationrelationship between the object in the real space and the target objecton the basis of the information acquired from the recognition processingunit 101. Furthermore, at this time, as described above, the calculationunit 103 may determine or update the position, orientation, movement, orthe like of the target object in consideration of the position ororientation of the viewpoint.

Then, the calculation unit 103 outputs information regarding thedetermined or updated position, orientation, movement, and the like(hereinafter, also referred to as “scene graph” for convenience) of theobject to the drawing processing unit 105 located in the subsequentstage. Furthermore, the calculation unit 103 provides at leastinformation regarding the determined or updated movement of the objectand information regarding the timing at which the movement of the objecthas been determined (in other words, the timing at which the scene graphhas been determined) of the information regarding the determined orupdated scene graph to the correction processing unit 107 to bedescribed below.

The drawing processing unit 105 acquires the information regarding thedetermined or updated scene graph from the calculation unit 103.Furthermore, the drawing processing unit 105 acquires the informationregarding the recognition result of the position or orientation of theviewpoint (for example, the information regarding the result ofself-position estimation) from the recognition processing unit 101. Thedrawing processing unit 105 draws the target object as the displayinformation in a predetermined buffer (for example, frame buffer)according to the position or orientation of the viewpoint based on theinformation acquired from the recognition processing unit 101 and theposition, orientation, movement, and the like of the object based on theinformation acquired from the calculation unit 103. Specifically, thedrawing processing unit 105 projects the target object (for example, thevirtual object having three-dimensional information) in the displayregion according to the relative position or orientation relationshipbetween the viewpoint and the object. Then, the drawing processing unit105 draws the display information (that is, two-dimensional displayinformation) according to the result of the projection in the buffer.

Furthermore, the drawing processing unit 105 may acquire informationregarding a recognition result of the position of an object in the realspace from the recognition processing unit 101. In this case, forexample, the drawing processing unit 105 may draw the displayinformation according to the recognition result of the object (realobject) in the real space. As a specific example, the drawing processingunit 105 may project an object to be presented onto the display regionaccording to the relative position or orientation relationship betweenthe object in the real space and the object, and draw displayinformation according to a result of the projection in the buffer.

Furthermore, the drawing processing unit 105 may control the color andbrightness of the display information drawn in the buffer according tovarious conditions. As a specific example, when projecting a targetobject on the display region, the drawing processing unit 105 maycontrol the color and brightness of the object to be drawn as thedisplay information according to a positional relationship between alight source defined in a three-dimensional space and the object.Furthermore, in a case of presenting the display information to besuperimposed on an object in the real space on the basis of the ARtechnology, the drawing processing unit 105 may control the color andbrightness of the display information to be drawn according to thepositional relationship between the object and the display information.

The correction processing unit 107 applies various types of correctionto the display information drawn in the buffer by the drawing processingunit 105. For example, the correction processing unit 107 performscorrection (so-called reprojection) of the display information accordingto the change in the position or orientation of the viewpoint until thedisplay information drawn in the buffer by the drawing processing unit105 is presented to the output unit 211 by the output control unit 109to be described below. In this case, the correction processing unit 107may acquire the information regarding the recognition result of theposition or orientation of the viewpoint from the recognition processingunit 101, for example, and use the information for the correction.

Furthermore, the correction processing unit 107 may control thecorrection in consideration of the movement of the object. Furthermore,the correction processing unit 107 acquires at least the informationregarding the movement of the object (for example, the informationregarding the moving direction and speed of the object) and theinformation regarding the timing at which the movement of the object hasbeen determined (in other words, the timing at which the scene graph hasbeen determined) of the information regarding the determined or updatedscene graph from the calculation unit 103. Note that the informationregarding the movement of the object corresponds to an example of “firstinformation”. Furthermore, the timing at which the movement of theobject has been determined corresponds to an example of “first timing”and the information regarding the timing corresponds to an example of“second information”.

Next, the correction processing unit 107 calculates the delay dt betweenthe timing at which the movement of the object is determined and thetiming at which execution of the correction is started, as describedabove with reference to FIG. 8 . Note that the timing at which executionof the correction is started corresponds to an example of “secondtiming”.

Then, the correction processing unit 107 presumes the changes accordingto the movement of the object (for example, the changes in thepresentation position, orientation, size, color, brightness, and thelike of the object) on the basis of the information regarding themovement of the object and the calculation result of the delay dt, andcontrols the correction of the display information according to a resultof the presumption.

Here, an example of a method of calculating the presentation position ofthe display information after correction in a case where the correctionprocessing unit 107 corrects the presentation position of the displayinformation will be described. Note that, in the present description, tomore simplify the description, an example of a method of calculating thepresentation position of an object after correction will be describedassuming that the object moves in an x direction (for example, thehorizontal direction as viewed from the viewpoint). Specifically, apresentation position x′ of the object after correction considering thedelay dt is expressed by the calculation formula given as the expression(2) below, where the presentation position of the object beforecorrection is x, and the speed of the object along the x direction is v.[Math. 2]x′=x+v×t  (2)

Note that the above expression (2) expresses the case where the objectis performing a constant velocity linear motion in the x direction, butcorrection content and parameters used for the correction can beappropriately changed according to the movement of the object.

As a specific example, in a case where the speed of the object changes,the correction processing unit 107 is only required to acquire theinformation regarding a change in the speed (that is, acceleration) ofthe object as the information regarding the movement of the object fromthe calculation unit 103, for example. Furthermore, in a case oftwo-dimensionally correcting the presentation position of the displayinformation, the correction processing unit 107 is only required toacquire information regarding two-dimensional movement of the object andinformation indicating the movement of the object by vectors as theinformation regarding the movement of the object from the calculationunit 103, for example.

Furthermore, as described above, the above correction is not necessarilylimited to the correction of the presentation position of the displayinformation, and for example, the orientation of the displayinformation, the size of the display information, the color of thedisplay information, the brightness of the display information, or thelike may be corrected, for example. As a specific example, in the caseof correcting the orientation of the display information, the correctionprocessing unit 107 is only required to acquire information regarding arotating direction of the object or information regarding the speed ofrotation (for example, angular velocity) as the information regardingthe movement of the object from the calculation unit 103, for example.Furthermore, in the case of correcting the size of the displayinformation, the correction processing unit 107 is only required toacquire information that can be a factor of enlargement or reduction ofthe display information according to the projection result of the objectas the information regarding the movement of the object from thecalculation unit 103. As a specific example, in a case where the displayinformation is visually recognized to be enlarged as the objectapproaches the viewpoint, the correction processing unit 107 is onlyrequired to acquire information regarding the speed of the objectapproaching the viewpoint (that is, the speed of the movement in a depthdirection of the object) from the calculation unit 103, for example.Furthermore, in a case where the object itself is enlarged or reduced,the correction processing unit 107 is only required to acquireinformation regarding the speed at which the object is enlarged orreduced from the calculation unit 103, for example. Furthermore, in thecase of correcting the color or brightness of the object, the correctionprocessing unit 107 is only required to acquire information regarding achange in the color or brightness of the object as the informationregarding the movement of the object from the calculation unit 103. Notethat examples of the information regarding the change in the color orbrightness of the object include information regarding a direction inwhich the color or brightness changes, and information regarding thespeed of the change. Furthermore, as another example, information thatcan be a factor for the change in the color or brightness of the object,such as information regarding a change in a positional relationshipbetween the object and a light source, or information regarding a changein light intensity from the light source may be used.

When various corrections have been performed for the display informationdrawn in the buffer by the correction processing unit 107 as describedabove, the output control unit 109 causes the output unit 211 to displaythe corrected display information held in the buffer in time with thedisplay timing of the information in the output unit 211.

Note that, for example, the processing block (that is, the processingblock including the drawing processing unit 105, the correctionprocessing unit 107, and the output control unit 109) illustrated withreference code 111, in the functional configuration of the informationprocessing system 1 illustrated in FIG. 10 , corresponds to an exampleof “control unit” that causes the display information to be displayed inthe display region of the output unit 211. Furthermore, the portion thatacquires the information regarding the movement of the viewpoint fromthe recognition processing unit 101 and the portion that acquires theinformation regarding the movement of the object from the calculationunit 103, of the processing block 111, correspond to an example of“acquisition unit”.

Note that the functional configurations of the information processingsystem 1 illustrated in FIG. 10 are mere examples, and the functionalconfigurations of the information processing system 1 are notnecessarily limited to the example illustrated in FIG. 10 as long as theoperation of the above-described configurations can be implemented. As aspecific example, at least one of the imaging unit 201, the detectionunit 251, or the output unit 211 may be integrally configured with theinformation processing apparatus 10. Furthermore, as another example,some functions of the information processing apparatus 10 may beprovided outside the information processing apparatus 10. As a specificexample, the portion corresponding to the recognition processing unit101 or the calculation unit 103 may be provided outside the informationprocessing apparatus 10. Note that, in this case, an interface thatacquires information from the portion corresponding to the recognitionprocessing unit 101 or the calculation unit 103 provided outside theinformation processing apparatus 10 can correspond to an example of the“acquisition unit”. Furthermore, at least some functions of theinformation processing apparatus 10 may be implemented by a plurality ofdevices operating in cooperation with one another.

An example of a functional configuration of the information processingsystem according to the embodiment of the present disclosure has beendescribed with reference to FIG. 10 .

<3.3. Processing>

Next, an example of a flow of the series of processing of theinformation processing system 1 according to the embodiment of thepresent disclosure will be described, in particular, focusing on theoperation of the information processing apparatus 10 illustrated in FIG.1 . For example, FIG. 11 is a flowchart illustrating an example of aflow of the series of processing of the information processing system 1according to the present embodiment, and illustrates an example of aflow of the processing in which the information processing apparatus 10causes the display information to be presented in the display regionaccording to the recognition result of the position or orientation ofthe viewpoint.

As illustrated in FIG. 11 , the information processing apparatus 10(recognition processing unit 101) recognizes at least one of theposition or orientation of the viewpoint on the basis of the informationregarding the imaging result of the image by the imaging unit 201 andthe detection result of the change in the position or orientation of theinput/output device 20 by the detection unit 251. The self-positionestimation technology such as SLAM may be used for the recognition ofthe position or orientation of the viewpoint, for example. As describedabove, the information processing apparatus 10 can recognize themovement of the viewpoint (that is, the change in the position andorientation of the viewpoint) by sequentially recognizing the positionand orientation of the viewpoint (S101).

Next, the information processing apparatus 10 (calculation unit 103)determines or updates the scene graph (in particular, determines orupdates the movement of the object) with the execution of apredetermined function such as an application. As a more specificexample, the information processing apparatus 10 may determine or updatethe position, orientation, movement, and the like of the target objectaccording to the relative position or orientation relationship betweenthe viewpoint and the object on the basis of the recognition result ofthe position or orientation of the viewpoint and autonomous movement ofthe object (S103).

Next, the information processing apparatus 10 (drawing processing unit105) draws the target object in a predetermined buffer (for example, aframe buffer) as the display information on the basis of the result ofthe determination or update of the scene graph and the recognitionresult of the position or orientation of the viewpoint. Specifically,the information processing apparatus 10 projects the target object (forexample, the virtual object having three-dimensional information) in thedisplay region according to the relative position or orientationrelationship between the viewpoint and the object. Then, the informationprocessing apparatus 10 draws display information (that is,two-dimensional display information) according to the result of theprojection in the buffer (S105).

Next, the information processing apparatus 10 (correction processingunit 107) performs correction (so-called reprojection) of the displayinformation drawn in the buffer according to the change in the positionor orientation of the viewpoint until the display information ispresented to the output unit 211. At this time, the informationprocessing apparatus 10 may control the correction in consideration ofthe movement of the object.

Specifically, the information processing apparatus 10 calculates a timedifference (that is, the delay dt) between the timing at which themovement of the object has been determined and the timing at whichexecution of the correction is started. Furthermore, the informationprocessing apparatus 10 presumes the changes according to the movementof the object (for example, corresponding to the changes in thepresentation position, orientation, size, and the like of the object, inother words, the change in the scene) on the basis of the informationregarding the movement of the object and the calculation result of thetime difference (S107). Then, the information processing apparatus 10controls the correction of the display information according to theresumption result of the change in the scene (S109).

Then, the information processing apparatus 10 (output control unit 109)causes the output unit 211 to display the corrected display informationheld in the buffer in time with the display timing of information to theoutput unit 211 (for example, the light emission timing of the outputunit 211). As a result, the display information is presented to the uservia the output unit 211 (S111).

As described above, the information processing apparatus 10 sequentiallyexecutes the series of processing illustrated with reference codes S101to S111 at predetermined periods unless an instruction on termination ofthe series of processing is given (S113, NO). Then, when receiving theinstruction on termination of execution of the series of processing(S113, YES), the information processing apparatus 10 terminates theexecution of the series of processing illustrated with reference codesS101 to S111.

Note that, part of the processing illustrated with reference codes S101to S111 may be executed in parallel with another processing. As aspecific example, the processing regarding recognition of the positionor orientation of the viewpoint illustrated with reference code S101 canbe sequentially executed even while the processing illustrated withreference codes S103 to S111 is being executed. As a result, the latestinformation can be used for the processing as the information regardingthe position or orientation of the viewpoint at the execution timing ofthe processing (in particular, the processing S103, S105, and S107)illustrated with reference codes S103 to S111.

An example of the flow of the series of processing of the informationprocessing system 1 according to the embodiment of the presentdisclosure has been described with reference to FIG. 11 , in particular,focusing on the operation of the information processing apparatus 10illustrated in FIG. 1 .

<3.4. Modification>

Next, modifications of the information processing system according tothe embodiment of the present disclosure will be described.

(Modification 1: An Example of Correction Control)

First, as modification 1, an example of the control of the correction(reprojection) of the display information according to the movement ofan object will be described. The delay dt described with reference toFIG. 8 sometimes becomes larger due to a temporary increase in the loadof at least a part of the processing described as “Scene update” and“drawing”, for example. In such a situation where the delay dt becomeslarger, an error of the presumption result of the changes according tothe movement of the object (for example, the changes in the position,orientation, size, color, brightness, and the like of the object)sometimes becomes larger. As a specific example, when the delay dtbecomes larger in a situation where a locus of the movement of theobject has a curvature, there may be a gap between the original locus ofthe movement of the object and the presumed locus of the movement of theobject. Furthermore, as another example, in a case where the movement ofthe object suddenly changes, so-called overshoot or the like occurs, andthere may be a gap between the original locus of the movement of theobject and the presumed locus of the movement of the object. There is apossibility that the influence associated with the sudden change in themovement of the object becomes larger as the delay dt becomes larger,for example.

In view of such a situation, the processing regarding correction of thedisplay information may be temporarily suppressed in the case where thecalculation result of the delay dt has suddenly changed, for example.

As a specific example, application of the correction of the displayinformation may be temporarily suppressed in a case where a differencebetween the delays dt calculated at different timings from each otherhas exceeded a threshold. With such control, in a case where thecalculation result of the delay dt temporarily increases with anincrease in the processing load or the like, and the application amountof the correction of the display information temporarily increases,application of the correction is suppressed, so that application of thecorrection by which the movement of the display information (that is,the movement of the object) suddenly changes can be suppressed, forexample.

Furthermore, as another example, a moving average dt_avg of the delaysdt may be used at the time of correcting the display information. Morespecifically, at the time of correcting the display information, thecorrection may be controlled on the basis of a difference between thedelay dt calculated at the timing of the correction and the movingaverage dt_avg of the delays dt up to the timing (that is, dt−dt_avg).With such control, even in the case where the calculation result of thedelay dt temporarily increases with an increase in the processing loador the like, the application amount of the correction of the displayinformation is suppressed, and application of the correction by whichthe movement of the display information (that is, the movement of theobject) suddenly changes can be suppressed.

As the modification 1, an example of the control of the correction(reprojection) of the display information according to the movement ofan object has been described.

(Modification 2: An Example of Correction Application Unit)

Next, as modification 2, an example of an application unit forcorrecting display information will be described. In the above-describedembodiment, an example in which the correction of the displayinformation according to the projection result of the object is appliedfor each object has been described. In contrast, in the modification 2,an example of a case where the target object is divided into a pluralityof parts, and the correction of the display information according to theprojection result of the object is applied for each divided part will bedescribed.

As a specific example, the correction may be applied to the displayinformation according to the projection result of the object for eachunit data (for example, pixels) constituting the display information. Inthis case, the information regarding the movement of the target object(that is, information regarding movement of a part corresponding to thepixels constituting the display information) may be acquired for eachpart (for example, a voxel) of the object. Thereby, the correction(reprojection) can be applied for each pixels constituting the displayinformation.

Furthermore, as another example, the information regarding the movementof the object (that is, information regarding movement of a partcorresponding to an edge) may be acquired for each edge of the object.In this case, for example, the correction (reprojection) may be appliedon the basis of the information regarding the movement of the objectacquired for the edge, for the part corresponding to the edge of thetarget object of the display information according to the projectionresult of the object.

As described above, the movement of the object can be more finelyreproduced by dividing the target object into a plurality of parts, andapplying the correction of the display information according to theprojection result of the object for each divided part.

Note that the above-description is a mere example and does notnecessarily limit the operation of the information processing systemaccording to the modification 2. That is, the unit of dividing thetarget object (in other words, the unit for which the correction isapplied) is not particularly limited as long as the object can bedivided into a plurality of parts, and the correction of the displayinformation according to the projection result of the object can beapplied for each divided part.

An example of the application unit for correcting display informationhas been described as the modification 2.

4. Hardware Configuration

Next, an example of a hardware configuration of the informationprocessing apparatus 10 that configures the information processingsystem according to the present embodiment will be described.

<4.1. Configuration Example as Independently Operable Device>

First, an example of a hardware configuration of an informationprocessing apparatus 900 in a case where the configuration correspondingto the above-described information processing apparatus 10 isimplemented as an independently operable device such as a PC, asmartphone, or a server (which will be referred to as the “informationprocessing apparatus 900” for convenience) will be described in detailwith reference to FIG. 12 . FIG. 12 is a functional block diagramillustrating an example of a hardware configuration of the informationprocessing apparatus 900 configuring the information processing systemaccording to the embodiment of the present disclosure.

The information processing apparatus 900 configuring the informationprocessing system 1 according to the present embodiment mainly includesa CPU 901, a ROM 902, and a RAM 903. Furthermore, the informationprocessing apparatus 900 further includes a host bus 907, a bridge 909,an external bus 911, an interface 913, an input device 915, an outputdevice 917, a storage device 919, a drive 921, a connection port 923,and a communication device 925.

The CPU 901 functions as an arithmetic processing unit and a controldevice, and controls general operation or part thereof of theinformation processing apparatus 900 according to various programsrecorded in the ROM 902, the RAM 903, the storage device 919, or aremovable recording medium 927. The ROM 902 stores programs, arithmeticoperation parameters, and the like used by the CPU 901. The RAM 903primarily stores the programs used by the CPU 901, parameters thatappropriately change in execution of the programs, and the like. The CPU901, the ROM 902, and the RAM 903 are mutually connected by the host bus907 configured by an internal bus such as a CPU bus. Note that therecognition processing unit 101, the calculation unit 103, the drawingprocessing unit 105, the correction processing unit 107, and the outputcontrol unit 109 described with reference to FIG. 11 can be implementedby the CPU 901, for example.

The host bus 907 is connected to the external bus 911 such as aperipheral component interconnect/interface (PCI) bus via the bridge909. Furthermore, the input device 915, the output device 917, thestorage device 919, the drive 921, the connection port 923, and thecommunication device 925 are connected to the external bus 911 via theinterface 913.

The input device 915 is an operation unit operated by the user, such asa mouse, a keyboard, a touch panel, a button, a switch, a lever, and apedal, for example. Furthermore, the input device 915 may be, forexample, a remote control unit (so-called remote controller) usinginfrared rays or other radio waves or an externally connected device 929such as a mobile phone or a PDA corresponding to an operation of theinformation processing apparatus 900. Moreover, the input device 915 isconfigured by, for example, an input control circuit for generating aninput signal on the basis of information input by the user using theabove-described operation unit and outputting the input signal to theCPU 901, or the like. The user of the information processing apparatus900 can input various data and give an instruction on processingoperations to the information processing apparatus 900 by operating theinput device 915.

The output device 917 is configured by a device that can visually oraudibly notify the user of acquired information. Examples of suchdevices include display devices such as a CRT display device, a liquidcrystal display device, a plasma display device, an EL display device, alamp, and the like, sound output devices such as a speaker and aheadphone, and a printer device. The output device 917 outputs, forexample, results obtained by various types of processing performed bythe information processing apparatus 900. Specifically, the displaydevice displays the results of the various types of processing performedby the information processing apparatus 900 as texts or images.Meanwhile, the sound output device converts an audio signal includingreproduced sound data, voice data, or the like into an analog signal andoutputs the analog signal. Note that the output unit 211 described withreference to FIG. 11 can be implemented by the output device 917, forexample.

The storage device 919 is a device for data storage configured as anexample of a storage unit of the information processing apparatus 900.The storage device 919 is configured by a magnetic storage device suchas a hard disk drive (HDD), a semiconductor storage device, an opticalstorage device, a magneto-optical storage device, or the like, forexample. The storage device 919 stores programs executed by the CPU 901,various data, and the like.

The drive 921 is a reader/writer for a recording medium, and is built inor is externally attached to the information processing apparatus 900.The drive 921 reads out information recorded on the removable recordingmedium 927 such as a mounted magnetic disk, optical disk,magneto-optical disk, or semiconductor memory, and outputs theinformation to the RAM 903. Furthermore, the drive 921 can also write arecord on the removable recording medium 927 such as the mountedmagnetic disk, optical disk, magneto-optical disk, or semiconductormemory. The removable recording medium 927 is, for example, a DVDmedium, an HD-DVD medium, a Blu-ray (registered trademark) medium, orthe like. Furthermore, the removable recording medium 927 may be acompact flash (CF (registered trademark)), a flash memory, a securedigital (SD) memory card, or the like. Furthermore, the removablerecording medium 927 may be, for example, an integrated circuit (IC)card on which a non-contact IC chip is mounted, an electronic device, orthe like.

The connection port 923 is a port for being directly connected to theinformation processing apparatus 900. Examples of the connection port923 include a universal serial bus (USB) port, an IEEE 1394 port, asmall computer system interface (SCSI) port, and the like. Otherexamples of the connection port 923 include an RS-232C port, an opticalaudio terminal, a high-definition multimedia interface (HDMI)(registered trademark) port, and the like. By connecting the externallyconnected device 929 to the connection port 923, the informationprocessing apparatus 900 directly acquires various data from theexternally connected device 929 and provides various data to theexternally connected device 929.

The communication device 925 is, for example, a communication interfaceconfigured by a communication device for being connected to acommunication network (network) 931, and the like. The communicationdevice 925 is, for example, a communication card for a wired or wirelesslocal area network (LAN), Bluetooth (registered trademark), a wirelessUSB (WUSB), or the like. Furthermore, the communication device 925 maybe a router for optical communication, a router for an asymmetricdigital subscriber line (ADSL), a modem for various communications, orthe like. The communication device 925 can transmit and receive signalsand the like, for example, to and from the Internet and othercommunication devices in accordance with a predetermined protocol suchas TCP/IP, for example. Furthermore, the communication network 931connected to the communication device 925 is configured by a network orthe like connected by wire or wirelessly, and may be, for example, theInternet, home LAN, infrared communication, radio wave communication,satellite communication, or the like.

An example of the hardware configuration that can implement thefunctions of the information processing apparatus 900 that configuresthe information processing system 1 according to the embodiment of thepresent disclosure has been described. Each of the above-describedconfiguration elements may be configured using general-purpose membersor may be configured by hardware specialized for the function of eachconfiguration element. Therefore, the hardware configuration to be usedcan be changed as appropriate according to the technical level of thetime of carrying out the present embodiment. Note that variousconfigurations corresponding to the information processing apparatus 900configuring the information processing system 1 according to the presentembodiment are naturally provided although not illustrated in FIG. 12 .

Note that a computer program for implementing the functions of theinformation processing apparatus 900 configuring the informationprocessing system 1 according to the above-described present embodimentcan be prepared and implemented on a personal computer or the like.Furthermore, a computer-readable recording medium in which such acomputer program is stored can be provided. The recording medium is, forexample, a magnetic disk, an optical disk, a magneto-optical disk, aflash memory, or the like. Furthermore, the above computer program maybe delivered via, for example, a network without using a recordingmedium. Furthermore, the number of computers that execute the computerprogram is not particularly limited. For example, a plurality ofcomputers (for example, a plurality of servers or the like) may executethe computer program in cooperation with one another. Note that a singlecomputer or a plurality of computers cooperating with one another isalso referred to as a “computer system”.

An example of the hardware configuration of the information processingapparatus 900 in the case of implementing the configurationcorresponding to the above-described information processing apparatus 10as the independently operable information processing apparatus 900 suchas a PC, a smartphone, or a server has been described in detail withreference to FIG. 12 .

<4.2. Configuration Example when Implementing Information ProcessingApparatus as Chip>

Next, an example of a hardware configuration of a chip 950 in the caseof implementing the configuration corresponding to the above-describedinformation processing apparatus 10 as a chip such as a GPU (which willbe referred to as the “chip 950” for convenience) will be described indetail with reference to FIG. 13 . FIG. 13 is a functional block diagramillustrating an example of a hardware configuration in the case ofimplementing the information processing apparatus configuring theinformation processing system according to the embodiment of the presentdisclosure as a chip.

As illustrated in FIG. 13 , the chip 950 includes an image processingunit (graphics and compute array: GCA) 951, a storage device (graphicsmemory controller: GMC) 953, a display interface (DIF) 955, a businterface (BIF) 957, a power control unit (power management unit: PMU)961, and a boot control unit (VGABIOS) 963. Furthermore, a compressionprocessing unit (compression unit) 959 may be interposed between theimage processing unit 951 and the storage device 953.

The image processing unit 951 corresponds to a processor that executesvarious types of processing regarding image processing. As a specificexample, the image processing unit 951 executes various types ofcalculation processing such as the above-described processing regardingprojecting an object, processing regarding drawing the displayinformation according to the projection result, and processing regardingcorrecting the display information such as reprojection. Furthermore, atthis time, the image processing unit 951 may read data stored in thestorage device 953 and use the data for execution of the various typesof arithmetic processing. Note that the processing of the recognitionprocessing unit 101, the calculation unit 103, the drawing processingunit 105, the correction processing unit 107, and the output controlunit 109, which has been described with reference to FIG. 11 , can beimplemented by the calculation processing by the image processing unit951, for example.

The storage device 953 is a configuration for temporarily or permanentlystoring various data. As a specific example, the storage device 953 maystore data according to execution results of the various types ofarithmetic processing by the image processing unit 951. The storagedevice 953 can be implemented on the basis of a technology of video RAM(VRAM), window RAM (WRAM), multibank DRAM (MDRAM), double-data-rate(DDR), graphics DDR (GDDR), high bandwidth memory (HBM), or the like,for example.

The compression processing unit 959 compresses and decompresses variousdata. As a specific example, the compression processing unit 959 maycompress data according to a calculation result by the image processingunit 951 when the data is stored in the storage device 953. Furthermore,when the image processing unit 951 reads data stored in the storagedevice 953, the compression processing unit 959 may decompress the datain the case where the data is compressed.

The display interface 955 is an interface for the chip 950 to send andreceive data to and from a display (for example, the output unit 211illustrated in FIG. 11 ). As a specific example, a result of drawing ofthe display information by the image processing unit 951 is output tothe display via the display interface 955. Further, as another example,in the case where the result of drawing of the display information bythe image processing unit 951 is stored in the storage device 953, theresult of the drawing stored in the storage device 953 is output to thedisplay via the display interface 955.

The bus interface 957 is an interface for the chip 950 to send andreceive data to and from other devices and external devices. As aspecific example, the data stored in the storage device 953 istransmitted to another device or an external device via the businterface 957. Furthermore, data transmitted from another device or anexternal device is input to the chip 950 via the bus interface 957. Notethat the data input to the chip 950 is stored in the storage device 953,for example.

The power control unit 961 is a configuration for controlling supply ofpower to each part of the chip 950.

The boot control unit 963 is a configuration for managing andcontrolling various types of processing related to boot, input/output ofvarious types of information, and the like at the time of booting thechip 950. The boot control unit 963 corresponds to a so-called videographics array basic input/output system (VGABIOS).

An example of the hardware configuration of the chip 950 in the case ofimplementing the configuration corresponding to the above-describedinformation processing apparatus 10 as the chip 950 such as a GPU hasbeen described in detail with reference to FIG. 13 . Note that, in thepresent disclosure, the case of implementing the configuration as the“information processing apparatus” can include the case of implementingthe configuration as a chip (in other words, components) to beincorporated in a device, as illustrated in FIG. 13 , in addition to thecase of implementing the configuration as one device, as illustrated inFIG. 12 .

5. Conclusion

As described above, in the information processing system according to anembodiment of the present disclosure, the information processingapparatus includes the acquisition unit that acquires the informationregarding the movement of the object, and the control unit that correctsthe display information according to the projection result of theobject. The control unit projects the object in the display region atthe projection timing set according to the timing for each first period,and corrects the display information according to the result of theprojection in accordance with the display timing set for each secondperiod shorter than the first period. Furthermore, the control unitcontrols the correction of the second display information to maintainthe continuity according to the movement of the object between the firstdisplay information displayed according to the first projection resultof the object in accordance with the first display timing and the seconddisplay information displayed according to the second projection resultof the object in accordance with the second display timing immediatelyafter the first display timing.

With the above configuration, the information processing systemaccording to the present embodiment can suppress the situation in whichthe display information is presented as if a portion where the movementof the object is discontinuous has occurred from becoming apparent inpresenting the object having movement as the display information. Thatis, the information processing system according to an embodiment of thepresent disclosure can present a video in which the movement of anobject to be presented is shown in a more natural manner, even under thesituation where the object has movement.

Note that the above description has focused on the case where thetechnology according to the embodiment of the present disclosure isapplied to the head-mounted device to which the transmission-typedisplay is applied as the output unit. However, the application of thetechnology is not necessarily limited. That is, the technique accordingto the present disclosure can be applied in the case of complementingthe movement of the object by the correction of the display informationin a situation where the position or orientation of the viewpoint andthe autonomous movement of the object may irregularly change, or in asituation where the length of the processing of drawing the object asthe display information may irregularly change.

As a more specific example, in the above-described embodiment, thedescription has been given focusing on the case where the information ismainly presented on the basis of the AR technology. However, a blur ofthe video (in other words, the situation where the display informationis presented as if a portion where the movement of the object isdiscontinuous has occurred) can be suppressed from becoming apparent,even in a case where the information is presented on the basis of a VRtechnology by applying the present technology. Furthermore, as anotherexample, the blur of the video can be suppressed from becoming apparentby applying the present technology even in a situation where upsamplingis performed when the video such as a moving image transferred via anetwork is presented via a television receiver.

Note that, as the situation where the autonomous movement of the objectmay irregularly change, a situation in which some information of a videolacks due to a load of the network in the situation where the video istransferred via the network, and the autonomous movement of the objectirregularly changes accordingly, can be assumed, for example.Furthermore, as the situation where the length of the processing ofdrawing the object as the display information may irregularly change, asituation where a frame is dropped with an increase in a processing loadof a device on the reproduction side in a case where a codec by which aload on decoding of a moving image can vary is applied, can be assumed,for example.

Although the favorable embodiment of the present disclosure has beendescribed in detail with reference to the accompanying drawings, thetechnical scope of the present disclosure is not limited to suchexamples. It is obvious that persons having ordinary knowledge in thetechnical field of the present disclosure can conceive variousmodifications or alterations within the scope of the technical ideadescribed in the claims, and the modifications and alterations arenaturally understood to belong to the technical scope of the presentdisclosure.

Furthermore, the effects described in the present specification aremerely illustrative or exemplary and are not restrictive. That is, thetechnology according to the present disclosure can exhibit other effectsobvious to those skilled in the art from the description of the presentspecification together with or in place of the above-described effects.

Note that following configurations also belong to the technical scope ofthe present disclosure.

(1)

An information processing apparatus including:

an acquisition unit configured to acquire information regarding movementof an object; and

a control unit configured to project the object in a display region at aprojection timing set according to a first period, and correct displayinformation according to a result of the projection in accordance withdisplay timings each set for each second period shorter than the firstperiod, in which

the control unit

controls correction of second display information to maintain continuityaccording to the movement of the object between

first display information displayed according to a first projectionresult of the object in accordance with a first display timing, and

the second display information displayed according to a secondprojection result of the object in accordance with a second displaytiming immediately after the first display timing.

(2)

The information processing apparatus according to (1), in which thecontrol unit controls the correction of the second display informationsuch that a difference between the first display information and thesecond display information substantially coincides with a change in acomponent related to the correction among changes according to themovement of the object between the first display timing and the seconddisplay timing.

(3)

The information processing apparatus according to (2), in which thecomponent related to the correction includes at least one of a positionof the display information, an orientation of the display information,or a size of the display information.

(4)

The information processing apparatus according to (2) or (3), in which

the control unit

corrects a position of the second display information in the displayregion such that

a change in a position between the first display information and thesecond display information in the display region, and

a change in a position of the object with respect to a viewpoint betweenthe first display timing and the second display timing according to themovement of the object

substantially coincide with each other.

(5)

The information processing apparatus according to any one of (1) to (4),in which

the control unit

controls the correction of the second display information on the basisof

a delay according to a timing at which the movement of the objectcorresponding to the second projection result has been determined andthe second display timing, and

the movement of the object.

(6)

The information processing apparatus according to (5), in which

the delay

is a period according to

the timing at which the movement of the object corresponding to thesecond projection result has been determined and

a timing at which the correction is applied to the second displayinformation.

(7)

The information processing apparatus according to (6), in which

the control unit

calculates a moving average of the delay determined for each timing atwhich the correction is applied, and

controls the correction of the second display information on the basisof a difference between the delay and a calculation result of the movingaverage.

(8)

The information processing apparatus according to (6) or (7), in whichthe control unit suppresses application of the correction in a casewhere a different between the delays determined for different timingsfrom each other among the timings at which the correction is appliedexceeds a threshold.

(9)

The information processing apparatus according to any one of (5) to (8),in which the movement of the object is determined according to apositional relationship between a viewpoint and the object.

(10)

The information processing apparatus according to any one of (5) to (9),in which the timing at which the movement of the object has beendetermined is a timing at which a scene graph regarding the object isdetermined.

(11)

The information processing apparatus according to any one of (1) to(10), in which the control unit acquires the second projection result byprojecting the object in the display region at the projection timing setaccording to a timing after the first projection result has beenacquired among timings each set for each of the first periods.

(12)

The information processing apparatus according to any one of (1) to(11), in which the projection timing is set as a timing after processingregarding determination of the movement of the object is completed, theprocessing having been executed at a timing after the result of theprevious projection is acquired as a starting point among timings eachset for each of the first periods.

(13)

The information processing apparatus according to any one of (1) to(12), in which the information regarding movement of an object includesinformation regarding at least one of a moving direction, a speed, anacceleration, or an angular velocity of the object.

(14)

The information processing apparatus according to any one of (1) to(13), in which

the acquisition unit acquires the information regarding movement of theobject for each part of the object, and

the control unit controls the correction of the display informationaccording to a projection result of the object on the basis of theinformation corresponding to the part, for each part of the object.

(15)

The information processing apparatus according to (1), in which

the display region is a display region of a head-mounted display,

the object is a virtual object that autonomously moves in a real space,

the first period is a period regarding determination or update of ascene graph of an augmented reality application, and

the control unit performs reprojection processing for the virtual objectprojected in the display region, for each of the display timings, on thebasis of a change in position and orientation of a viewpoint of thehead-mounted display, as the correction of the display information.

(16)

The information processing apparatus according to (15), in which

the projection timing includes a first projection timing and a secondprojection timing after the first projection timing,

the first display information is display information corresponding tothe first projection timing,

the second display information is display information corresponding tothe second projection timing, and

the control unit determines a position of the second display informationon the basis of the first display information for which the reprojectionprocessing has been performed in accordance with the first displaytiming immediately before the second display timing.

(17)

The information processing apparatus according to (16), in which thecontrol unit determines the position of the second display informationon the basis of a timestamp of the first display information for whichthe reprojection processing has been performed.

(18)

An information processing method including:

by a computer,

acquiring information regarding movement of an object; and

projecting the object in a display region at a projection timing setaccording to a timing for each first period, and correcting displayinformation according to a result of the projection in accordance with adisplay timing set for each second period shorter than the first period,in which

correction of second display information is controlled to maintaincontinuity according to the movement of the object between

first display information displayed according to a first projectionresult of the object in accordance with a first display timing, and

the second display information displayed according to a secondprojection result of the object in accordance with a second displaytiming immediately after the first display timing.

(19)

A storage medium recording a program for causing a computer to execute:

acquiring information regarding movement of an object; and

projecting the object in a display region at a projection timing setaccording to a timing set for each first period, and correcting displayinformation according to a result of the projection in accordance with adisplay timing set for each second period shorter than the first period,in which

correction of second display information is controlled to maintaincontinuity according to the movement of the object between

first display information displayed according to a first projectionresult of the object in accordance with a first display timing, and

the second display information displayed according to a secondprojection result of the object in accordance with a second displaytiming immediately after the first display timing.

(20)

An information processing apparatus including:

an acquisition unit configured to acquire first information regardingmovement of an object, and second information regarding a first timingat which the movement of the object has been determined; and

a control unit configured to project the object to a display region onthe basis of the first information and the second information, andperform control so that display information is presented in the displayregion according to a result of the projection, in which

the control unit

controls correction of the display information on the basis of

a delay according to the first timing according to the secondinformation and a second timing at which the display information ispresented in the display region, and

the movement of the object according to the first information.

(21)

An information processing method including:

by a computer,

acquiring first information regarding movement of an object, and secondinformation regarding a first timing at which the movement of the objecthas been determined; and

projecting the object to a display region on the basis of the firstinformation and the second information, and performing control so thatdisplay information is presented in the display region according to aresult of the projection, in which

correction of the display information is controlled on the basis of

a delay according to the first timing according to the secondinformation and a second timing at which the display information ispresented in the display region, and

the movement of the object according to the first information.

(22)

A storage medium recording a program for causing a computer to execute:

acquiring first information regarding movement of an object, and secondinformation regarding a first timing at which the movement of the objecthas been determined; and

projecting the object to a display region on the basis of the firstinformation and the second information, and performing control so thatdisplay information is presented in the display region according to aresult of the projection, in which

correction of the display information is controlled on the basis of

a delay according to the first timing according to the secondinformation and a second timing at which the display information ispresented in the display region, and

the movement of the object according to the first information.

REFERENCE SIGNS LIST

-   1 Information processing system-   10 Information processing apparatus-   101 Recognition processing unit-   103 Calculation unit-   105 Drawing processing unit-   107 Correction processing unit-   109 Output control unit-   20 Input/output device-   201 Imaging unit-   207 Operation unit-   211 Output unit-   251 Detection unit

The invention claimed is:
 1. An information processing apparatuscomprising: an acquisition unit configured to acquire informationregarding movement of an object; and a control unit configured toproject the object in a display region at a projection timing setaccording to a first frequency, and correct display informationaccording to a result of the projection in accordance with a pluralityof display timings each set according to a second frequency larger thanthe first frequency, control correction of second display informationdisplayed at the second frequency to maintain continuity according tothe movement of the object between first display information displayedaccording to a first projection result of the object in accordance witha first display timing at the second frequency, the first projectionresult of the object being displayed according to the first frequency,and the second display information displayed according to a secondprojection result of the object in accordance with a second displaytiming at the second frequency immediately after the first displaytiming, the second projection result of the object being displayed atthe first frequency, and control the correction of the second displayinformation on a basis of a delay according to a timing at the firstfrequency at which the movement of the object corresponding to thesecond projection result has been determined and the second displaytiming at the second frequency, and the movement of the object, whereinthe acquisition unit and the control unit are each implemented via atleast one processor.
 2. The information processing apparatus accordingto claim 1, wherein the control unit is further configured to controlthe correction of the second display information such that a differencebetween the first display information and the second display informationsubstantially coincides with a change in a component related to thecorrection among changes according to the movement of the object betweenthe first display timing and the second display timing.
 3. Theinformation processing apparatus according to claim 2, wherein thecomponent related to the correction includes at least one of a positionof the display information, an orientation of the display information,or a size of the display information.
 4. The information processingapparatus according to claim 2, wherein the control unit is furtherconfigured to correct a position of the second display information inthe display region such that a change in a position between the firstdisplay information and the second display information in the displayregion, and a change in a position of the object with respect to aviewpoint between the first display timing and the second display timingaccording to the movement of the object substantially coincide with eachother.
 5. The information processing apparatus according to claim 1,wherein the delay is a period according to the timing at which themovement of the object corresponding to the second projection result hasbeen determined, and a timing at which the correction is applied to thesecond display information.
 6. The information processing apparatusaccording to claim 5, wherein the control unit is further configured tocalculate a moving average of the delay determined for each timing atwhich the correction is applied, and control the correction of thesecond display information on a basis of a difference between the delayand a calculation result of the moving average.
 7. The informationprocessing apparatus according to claim 5, wherein the control unit isfurther configured to suppress application of the correction in a casewhere a difference between the delays determined for different timingsfrom each other among the timings at which the correction is appliedexceeds a threshold.
 8. The information processing apparatus accordingto claim 1, wherein the movement of the object is determined accordingto a positional relationship between a viewpoint and the object.
 9. Theinformation processing apparatus according to claim 1, wherein thetiming at which the movement of the object has been determined is atiming at which a scene graph regarding the object is determined. 10.The information processing apparatus according to claim 1, wherein thecontrol unit is further configured to acquire the second projectionresult by projecting the object in the display region at the projectiontiming set according to a timing after the first projection result hasbeen acquired among timings each set according to the first frequency.11. The information processing apparatus according to claim 1, whereinthe projection timing is set as a timing after processing regardingdetermination of the movement of the object is completed, the processinghaving been executed at a timing after the result of a previousprojection is acquired as a starting point among timings each setaccording to the first frequency.
 12. The information processingapparatus according to claim 1, wherein the information regardingmovement of an object includes information regarding at least one of amoving direction, a speed, an acceleration, or an angular velocity ofthe object.
 13. The information processing apparatus according to claim1, wherein the acquisition unit is further configured to acquire theinformation regarding movement of the object for each part of theobject, and the control unit is further configured to control thecorrection of the display information according to a projection resultof the object on a basis of the information corresponding to the part,for each part of the object.
 14. The information processing apparatusaccording to claim 1, wherein the display region is a display region ofa head-mounted display, the object is a virtual object that autonomouslymoves in a real space, the first frequency is a frequency regardingdetermination or update of a scene graph of an augmented realityapplication, and the control unit is further configured to performreprojection processing for the virtual object projected in the displayregion, for each of the display timings, on a basis of a change inposition and orientation of a viewpoint of the head-mounted display, asthe correction of the display information.
 15. The informationprocessing apparatus according to claim 14, wherein the projectiontiming includes a first projection timing and a second projection timingafter the first projection timing, the first display informationincludes display information corresponding to the first projectiontiming, the second display information includes display informationcorresponding to the second projection timing, and the control unit isfurther configured to determine a position of the second displayinformation on a basis of the first display information for which thereprojection processing has been performed in accordance with the firstdisplay timing immediately before the second display timing.
 16. Theinformation processing apparatus according to claim 15, wherein thecontrol unit is further configured to determine the position of thesecond display information on a basis of a timestamp of the firstdisplay information for which the reprojection processing has beenperformed.
 17. An information processing method comprising: by acomputer, acquiring information regarding movement of an object;projecting the object in a display region at a projection timing setaccording to a first frequency, and correcting display informationaccording to a result of the projection in accordance with a pluralityof display timings each set according to a second frequency larger thanthe first frequency; controlling correction of second displayinformation displayed at the second frequency to maintain continuityaccording to the movement of the object between first displayinformation displayed according to a first projection result of theobject in accordance with a first display timing at the secondfrequency, the first projection result of the object being displayedaccording to the first frequency, and the second display informationdisplayed according to a second projection result of the objectdisplayed at the first frequency in accordance with a second displaytiming at the second frequency immediately after the first displaytiming, the second projection result of the object being displayed atthe first frequency; and controlling the correction of the seconddisplay information on a basis of a delay according to a timing at thefirst frequency at which the movement of the object corresponding to thesecond projection result has been determined and the second displaytiming at the second frequency, and the movement of the object.
 18. Anon-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer causes the computer toexecute an information processing method, the method comprising:acquiring information regarding movement of an object; projecting theobject in a display region at a projection timing set according to afirst frequency, and correcting display information according to aresult of the projection in accordance with a plurality of displaytimings each set according to a second frequency larger than the firstfrequency; controlling correction of second display informationdisplayed at the second frequency to maintain continuity according tothe movement of the object between first display information displayedaccording to a first projection result of the object in accordance witha first display timing at the second frequency, the first projectionresult of the object being displayed according to the first frequency,and the second display information displayed according to a secondprojection result of the object in accordance with a second displaytiming at the second frequency immediately after the first displaytiming, the second projection result of the object being displayed atthe first frequency; and controlling the correction of the seconddisplay information on a basis of a delay according to a timing at thefirst frequency at which the movement of the object corresponding to thesecond projection result has been determined and the second displaytiming at the second frequency, and the movement of the object.